{ "version": "https://jsonfeed.org/version/1", "title": "Saved for Later", "feed_url": "https://savedforlater.dev/json", "items": [ { "content_html": "Comments", "url": "https://www.anthropic.com/news/claude-design-anthropic-labs", "title": "Claude Design", "date_modified": "2026-04-17T15:04:09.000Z" }, { "content_html": "Comments", "url": "https://stagereview.app/", "title": "Show HN: Stage – Putting humans back in control of code review", "date_modified": "2026-04-16T17:36:29.000Z" }, { "content_html": "Comments", "url": "https://openai.com/index/codex-for-almost-everything/", "title": "Codex for Almost Everything", "date_modified": "2026-04-16T17:12:19.000Z" }, { "content_html": "Comments", "url": "https://openai.com/index/codex-for-almost-everything/", "title": "Codex for almost everything", "date_modified": "2026-04-16T17:12:19.000Z" }, { "content_html": "Comments", "url": "https://cirruslabs.org/", "title": "Cirrus Labs to join OpenAI", "date_modified": "2026-04-11T13:01:34.000Z" }, { "content_html": "Comments", "url": "https://blog.gitbutler.com/series-a", "title": "We've raised $17M to build what comes after Git", "date_modified": "2026-04-10T01:52:58.000Z" }, { "content_html": "Comments", "url": "https://www.yanist.com/clean-code-in-the-age-of-coding-agents/", "title": "Clean code in the age of coding agents", "date_modified": "2026-04-09T14:33:08.000Z" }, { "content_html": "Comments", "url": "https://astral.sh/blog/open-source-security-at-astral", "title": "Open Source Security at Astral", "date_modified": "2026-04-09T04:11:55.000Z" }, { "content_html": "Comments", "url": "https://astral.sh/blog/open-source-security-at-astral", "title": "Open source security at Astral", "date_modified": "2026-04-09T04:11:55.000Z" }, { "content_html": "Comments", "url": "https://lucumr.pocoo.org/2026/4/8/mario-and-earendil/", "title": "Mario and Earendil", "date_modified": "2026-04-08T09:13:17.000Z" }, { "content_html": "Comments", "url": "https://www.allthingsdistributed.com/2026/04/s3-files-and-the-changing-face-of-s3.html", "title": "S3 Files", "date_modified": "2026-04-07T19:44:01.000Z" }, { "content_html": "Comments", "url": "https://www.skyvern.com/blog/getting-claude-to-qa-its-own-work/", "title": "Getting Claude to QA its own work", "date_modified": "2026-04-03T17:18:38.000Z" }, { "content_html": "

Today, we’re announcing managed daemon support for Amazon Elastic Container Service (Amazon ECS) Managed Instances. This new capability extends the managed instances experience we introduced in September 2025, by giving platform engineers independent control over software agents such as monitoring, logging, and tracing tools, without requiring coordination with application development teams, while also improving reliability by ensuring every instance consistently runs required daemons and enabling comprehensive host-level monitoring.

\n

When running containerized workloads at scale, platform engineers manage a wide range of responsibilities, from scaling and patching infrastructure to keeping applications running reliably and maintaining the operational agents that support those applications. Until now, many of these concerns were tightly coupled. Updating a monitoring agent meant coordinating with application teams, modifying task definitions, and redeploying entire applications, a significant operational burden when you’re managing hundreds or thousands of services.

\n

Decoupled lifecycle management for daemons
Amazon ECS now introduces a dedicated managed daemons construct that enables platform teams to centrally manage operational tooling. This separation of concerns allows platform engineers to independently deploy and update monitoring, logging, and tracing agents to infrastructure, while enforcing consistent use of required tools across all instances, without requiring application teams to redeploy their services. Daemons are guaranteed to start before application tasks and drain last, ensuring that logging, tracing, and monitoring are always available when your application needs them.

\n

Platform engineers can deploy managed daemons across multiple capacity providers, or target specific capacity providers, giving them flexibility in how they roll out agents across their infrastructure. Resource management is also centralized, allowing teams to define daemon CPU and memory parameters separately from application configurations with no need to rebuild AMIs or update task definitions, while optimizing resource utilization since each instance runs exactly one daemon copy shared across multiple application tasks.

\n

Let’s try it out
To take ECS Managed Daemons for a spin, I decided to start with the Amazon CloudWatch Agent as my first managed daemon. I had previously set up an Amazon ECS cluster with a Managed Instance capacity provider using the documentation.

\n

From the Amazon Elastic Container Service console, I noticed a new Daemon task definitions option in the navigation pane, where I can define my managed daemons.

\n

\"Managed

\n

I chose Create new daemon task definition to get started. For this example, I configured the CloudWatch Agent with 1 vCPU and 0.5 GB of memory. In the Daemon task definition family field, I entered a name I’d recognize later.

\n

For the Task execution role, I selected ecsTaskExecutionRole from the dropdown. Under the Container section, I gave my container a descriptive name and pasted in the image URI: public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest along with a few additional details.

\n

After reviewing everything, I chose Create.

\n

Once my daemon task definition was created, I navigated to the Clusters page, selected my previously created cluster and found the new Daemons tab.

\n

\"Managed

\n

Here I can simply click the Create daemon button and complete the form to configure my daemon.

\n

\"Managed

\n

Under Daemon configuration, I selected my newly created daemon task definition family and then assigned my daemon a name. For Environment configuration, I selected the ECS Managed Instances capacity provider I had set up earlier. After confirming my settings, I chose Create.

\n

Now ECS automatically ensures the daemon task launches first on every provisioned ECS managed instance in my selected capacity provider. To see this in action, I deployed a sample nginx web service as a test workload. Once my workload was deployed, I could see in the console that ECS Managed Daemons had automatically deployed the CloudWatch Agent daemon alongside my application, with no manual intervention required.

\n

When I later updated my daemon, ECS handled the rolling deployment automatically by provisioning new instances with the updated daemon, starting the daemon first, then migrating application tasks to the new instances before terminating the old ones. This “start before stop” approach ensures continuous daemon coverage: your logging, monitoring, and tracing agents remain operational throughout the update with no gaps in data collection. The drain percentage I configured controlled the pace of this replacement, giving me complete control over addon updates without any application downtime.

\n

How it works
The managed daemon experience introduces a new daemon task definition that is separate from task definitions, with its own parameters and validation scheme. A new daemon_bridge network mode enables daemons to communicate with application tasks while remaining isolated from application networking configurations.

\n

Managed daemons support advanced host-level access capabilities that are essential for operational tooling. Platform engineers can configure daemon tasks as privileged containers, add additional Linux capabilities, and mount paths from the underlying host filesystem. These capabilities are particularly valuable for monitoring and security agents that require deep visibility into host-level metrics, processes, and system calls.

\n

When a daemon is deployed, ECS launches exactly one daemon process per container instance before placing application tasks. This guarantees that operational tooling is in place before your application starts receiving traffic. ECS also supports rolling deployments with automatic rollbacks, so you can update agents with confidence.

\n

Now available
Managed daemon support for Amazon ECS Managed Instances is available today in all AWS Regions. To get started, visit the Amazon ECS console or review the Amazon ECS documentation. You can also explore the new managed daemons Application Programming Interface (APIs) by visiting this website.

\n

There is no additional cost to use managed daemons. You pay only for the standard compute resources consumed by your daemon tasks.

", "url": "https://aws.amazon.com/blogs/aws/announcing-managed-daemon-support-for-amazon-ecs-managed-instances/", "title": "Announcing managed daemon support for Amazon ECS Managed Instances", "date_modified": "2026-04-01T23:31:24.000Z" }, { "content_html": "Comments", "url": "https://jeapostrophe.github.io/tech/jc-workflow/", "title": "Don't Wait for Claude", "date_modified": "2026-03-27T18:13:45.000Z" }, { "content_html": "Comments", "url": "https://lubeno.dev/blog/reinventing-the-pull-request", "title": "Reinventing the Pull Request", "date_modified": "2026-03-27T09:04:29.000Z" }, { "content_html": "Comments", "url": "https://david.alvarezrosa.com/posts/optimizing-a-lock-free-ring-buffer/", "title": "Optimizing a lock-free ring buffer", "date_modified": "2026-03-24T12:52:04.000Z" }, { "content_html": "Comments", "url": "https://github.com/antithesishq/bombadil", "title": "Bombadil: Property-based testing for web UIs by Antithesis", "date_modified": "2026-03-19T08:53:36.000Z" }, { "content_html": "Comments", "url": "https://github.com/antithesishq/bombadil", "title": "Bombadil: Property-based testing for web UIs", "date_modified": "2026-03-19T08:53:36.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/toddsaunders/status/2034243420147859716", "title": "An industrial piping contractor on Claude Code [video]", "date_modified": "2026-03-18T20:50:56.000Z" }, { "content_html": "

Comments

", "url": "https://andrealeopardi.com/posts/beam-metrics-in-clickhouse/", "title": "BEAM Metrics in ClickHouse", "date_modified": "2026-03-18T15:44:05.000Z" }, { "content_html": "Comments", "url": "https://trigger.dev/blog/how-trql-works", "title": "We give every user SQL access to a shared ClickHouse cluster", "date_modified": "2026-03-17T15:50:03.000Z" }, { "content_html": "Comments", "url": "https://apenwarr.ca/log/20260316", "title": "Every layer of review makes you 10x slower", "date_modified": "2026-03-17T03:20:36.000Z" }, { "content_html": "Comments", "url": "https://arxiv.org/abs/2511.04427", "title": "Speed at the cost of quality: Study of use of Cursor AI in open source projects", "date_modified": "2026-03-16T17:07:37.000Z" }, { "content_html": "Comments", "url": "https://towlion.github.io", "title": "An experiment to use GitHub Actions as a control plane for a PaaS", "date_modified": "2026-03-16T00:51:18.000Z" }, { "content_html": "Comments", "url": "https://digg.com/", "title": "Digg is gone again", "date_modified": "2026-03-13T18:52:17.000Z" }, { "content_html": "

Today, we’re announcing a new feature of Amazon Simple Storage Service (Amazon S3) you can use to create general purpose buckets in your own account regional namespace simplifying bucket creation and management as your data storage needs grow in size and scope. You can create general purpose bucket names across multiple AWS Regions with assurance that your desired bucket names will always be available for you to use.

\n

With this feature, you can predictably name and create general purpose buckets in your own account regional namespace by appending your account’s unique suffix in your requested bucket name. For example, I can create the bucket mybucket-123456789012-us-east-1-an in my account regional namespace. mybucket is the bucket name prefix that I specified, then I add my account regional suffix to the requested bucket name: -123456789012-us-east-1-an. If another account tries to create buckets using my account’s suffix, their requests will be automatically rejected.

\n

Your security teams can use AWS Identity and Access Management (AWS IAM) policies and AWS Organizations service control policies to enforce that your employees only create buckets in their account regional namespace using the new s3:x-amz-bucket-namespace condition key, helping teams adopt the account regional namespace across your organization.

\n

Create your S3 bucket with account regional namespace in action
To get started, choose Create bucket in the Amazon S3 console. To create your bucket in your account regional namespace, choose Account regional namespace. If you choose this option, you can create your bucket with any name that is unique to your account and region.

\n

This configuration supports all of the same features as general purpose buckets in the global namespace. The only difference is that only your account can use bucket names with your account’s suffix. The bucket name prefix and the account regional suffix combined must be between 3 and 63 characters long.

\n

\n

Using the AWS Command Line Interface (AWS CLI), you can create a bucket with account regional namespace by specifying the x-amz-bucket-namespace:account-regional request header and providing a compatible bucket name.

\n
$ aws s3api create-bucket --bucket mybucket-123456789012-us-east-1-an \\\n   --bucket-namespace account-regional \\\n   --region us-east-1
\n

You can use the AWS SDK for Python (Boto3) to create a bucket with account regional namespace using CreateBucket API request.

\n
import boto3\n\nclass AccountRegionalBucketCreator:\n    \"\"\"Creates S3 buckets using account-regional namespace feature.\"\"\"\n    \n    ACCOUNT_REGIONAL_SUFFIX = \"-an\"\n    \n    def __init__(self, s3_client, sts_client):\n        self.s3_client = s3_client\n        self.sts_client = sts_client\n    \n    def create_account_regional_bucket(self, prefix):\n        \"\"\"\n        Creates an account-regional S3 bucket with the specified prefix.\n        Resolves caller AWS account ID using the STS GetCallerIdentity API.\n        Format: ---an\n        \"\"\"\n        account_id = self.sts_client.get_caller_identity()['Account']\n        region = self.s3_client.meta.region_name\n        bucket_name = self._generate_account_regional_bucket_name(\n            prefix, account_id, region\n        )\n        \n        params = {\n            \"Bucket\": bucket_name,\n            \"BucketNamespace\": \"account-regional\"\n        }\n        if region != \"us-east-1\":\n            params[\"CreateBucketConfiguration\"] = {\n                \"LocationConstraint\": region\n            }\n        \n        return self.s3_client.create_bucket(**params)\n    \n    def _generate_account_regional_bucket_name(self, prefix, account_id, region):\n        return f\"{prefix}-{account_id}-{region}{self.ACCOUNT_REGIONAL_SUFFIX}\"\n\n\nif __name__ == '__main__':\n    s3_client = boto3.client('s3')\n    sts_client = boto3.client('sts')\n    \n    creator = AccountRegionalBucketCreator(s3_client, sts_client)\n    response = creator.create_account_regional_bucket('test-python-sdk')\n    \n    print(f\"Bucket created: {response}\")
\n

You can update your infrastructure as code (IaC) tools, such as AWS CloudFormation, to simplify creating buckets in your account regional namespace. AWS CloudFormation offers the pseudo parameters, AWS::AccountId and AWS::Region, making it easy to build CloudFormation templates that create account regional namespace buckets.

\n

The following example demonstrates how you can update your existing CloudFormation templates to start creating buckets in your account regional namespace:

\n
BucketName: !Sub \"amzn-s3-demo-bucket-${AWS::AccountId}-${AWS::Region}-an\"\nBucketNamespace: \"account-regional\"
\n

Alternatively, you can also use the BucketNamePrefix property to update your CloudFormation template. By using the BucketNamePrefix, you can provide only the customer defined portion of the bucket name and then it automatically adds the account regional namespace suffix based on the requesting AWS account and Region specified.

\n
BucketNamePrefix: 'amzn-s3-demo-bucket'\nBucketNamespace: \"account-regional\"\n
\n

Using these options, you can build a custom CloudFormation template to easily create general purpose buckets in your account regional namespace.

\n

Things to know
You can’t rename your existing global buckets to bucket names with account regional namespace, but you can create new general purpose buckets in your account regional namespace. Also, the account regional namespace is only supported for general purpose buckets. S3 table buckets and vector buckets already exist in an account-level namespace and S3 directory buckets exist in a zonal namespace.

\n

To learn more, visit Namespaces for general purpose buckets in the Amazon S3 User Guide.

\n

Now available
Creating general purpose buckets in your account regional namespace in Amazon S3 is now available in 37 AWS Regions including the AWS China and AWS GovCloud (US) Regions. You can create general purpose buckets in your account regional namespace at no additional cost.

\n

Give it a try in the Amazon S3 console today and send feedback to AWS re:Post for Amazon S3 or through your usual AWS Support contacts.

\n

— Channy

", "url": "https://aws.amazon.com/blogs/aws/introducing-account-regional-namespaces-for-amazon-s3-general-purpose-buckets/", "title": "Introducing account regional namespaces for Amazon S3 general purpose buckets", "date_modified": "2026-03-12T21:18:55.000Z" }, { "content_html": "Comments", "url": "https://www.mendral.com/blog/ci-at-scale", "title": "What CI looks like at a 100-person team (PostHog)", "date_modified": "2026-03-12T15:50:09.000Z" }, { "content_html": "Comments", "url": "https://www.proxylity.com/articles/wireguard-is-two-things.html", "title": "WireGuard Is Two Things", "date_modified": "2026-03-12T04:38:49.000Z" }, { "content_html": "Comments", "url": "https://modulus.so", "title": "Show HN: Modulus – Cross-repository knowledge orchestration for coding agents", "date_modified": "2026-03-10T18:52:03.000Z" }, { "content_html": "Comments", "url": "https://agent-safehouse.dev/", "title": "Agent Safehouse – macOS-native sandboxing for local agents", "date_modified": "2026-03-08T20:30:18.000Z" }, { "content_html": "Comments", "url": "https://github.com/gritzko/librdx/tree/master/be", "title": "Beagle, a source code management system that stores AST trees", "date_modified": "2026-03-08T13:28:26.000Z" }, { "content_html": "Comments", "url": "https://mksg.lu/blog/context-mode", "title": "MCP server that reduces Claude Code context consumption by 98%", "date_modified": "2026-02-28T10:01:20.000Z" }, { "content_html": "Comments", "url": "https://www.shayon.dev/post/2026/52/lets-discuss-sandbox-isolation/", "title": "Let's discuss sandbox isolation", "date_modified": "2026-02-27T18:49:50.000Z" }, { "content_html": "Comments", "url": "https://book.mixu.net/distsys/single-page.html", "title": "Distributed Systems for Fun and Profit", "date_modified": "2026-02-24T15:32:00.000Z" }, { "content_html": "Comments", "url": "https://trunkbaseddevelopment.com/", "title": "Trunk Based Development", "date_modified": "2026-02-21T07:07:12.000Z" }, { "content_html": "Comments", "url": "https://georgeguimaraes.com/your-agent-orchestrator-is-just-a-bad-clone-of-elixir/", "title": "Your Agent Framework Is Just a Bad Clone of Elixir", "date_modified": "2026-02-18T22:37:35.000Z" }, { "content_html": "Comments", "url": "https://haskellforall.com/2026/02/browse-code-by-meaning", "title": "Browse Code by Meaning", "date_modified": "2026-02-17T11:07:19.000Z" }, { "content_html": "Comments", "url": "https://www.bbc.com/news/articles/cx2gn239exlo", "title": "Dark web agent spotted bedroom wall clue to rescue girl from abuse", "date_modified": "2026-02-17T01:01:00.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/zero-downtime-migrations-at-petabyte-scale", "title": "Zero downtime migrations at Petabyte scale", "date_modified": "2026-02-16T17:35:00.000Z" }, { "content_html": "Comments", "url": "https://gist.github.com/gritzko/6e81b5391eacb585ae207f5e634db07e", "title": "Git is a file system. We need a database for the code", "date_modified": "2026-02-15T09:09:31.000Z" }, { "content_html": "Comments", "url": "https://gist.github.com/gritzko/6e81b5391eacb585ae207f5e634db07e", "title": "SCM as a database for the code", "date_modified": "2026-02-15T09:09:31.000Z" }, { "content_html": "Comments", "url": "https://monosketch.io/", "title": "Monosketch", "date_modified": "2026-02-13T12:18:05.000Z" }, { "content_html": "Comments", "url": "https://github.com/aws/aws-sdk-go-v2/commit/3dca5e45d5ad05460b93410087833cbaa624754e", "title": "AWS Adds support for nested virtualization", "date_modified": "2026-02-13T00:07:57.000Z" }, { "content_html": "Comments", "url": "https://entire.io/blog/hello-entire-world/", "title": "Ex-GitHub CEO Launches a New Developer Platform for AI Agents", "date_modified": "2026-02-10T15:44:47.000Z" }, { "content_html": "Comments", "url": "https://code.storage/", "title": "Code Storage by the Pierre Computer Company", "date_modified": "2026-02-10T10:12:27.000Z" }, { "content_html": "Comments", "url": "https://www.githubstatus.com/incidents/lcw3tg2f6zsd", "title": "Another GitHub outage in the same day", "date_modified": "2026-02-09T19:07:25.000Z" }, { "content_html": "Comments", "url": "https://simonwillison.net/2026/Feb/7/software-factory/", "title": "StrongDM's AI team build serious software without even looking at the code", "date_modified": "2026-02-07T15:41:05.000Z" }, { "content_html": "Comments", "url": "https://factory.strongdm.ai/", "title": "Software factories and the agentic moment", "date_modified": "2026-02-07T15:05:56.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/introducing-deno-sandbox", "title": "Deno Sandbox", "date_modified": "2026-02-03T17:33:20.000Z" }, { "content_html": "Comments", "url": "https://blog.rbby.dev/posts/github-ai-contribution-blame-for-pull-requests/", "title": "GitHub Browser Plugin for AI Contribution Blame in Pull Requests", "date_modified": "2026-02-03T14:35:25.000Z" }, { "content_html": "Comments", "url": "https://github.com/oug-t/difi", "title": "Show HN: difi – A Git diff TUI with Neovim integration (written in Go)", "date_modified": "2026-02-03T13:47:24.000Z" }, { "content_html": "Comments", "url": "https://www.githubstatus.com?todayis=2026-02-02", "title": "GitHub experience various partial-outages/degradations", "date_modified": "2026-02-02T21:28:16.000Z" }, { "content_html": "Comments", "url": "https://openai.com/index/introducing-the-codex-app/", "title": "The Codex App", "date_modified": "2026-02-02T18:02:48.000Z" }, { "content_html": "Comments", "url": "https://matklad.github.io/2026/01/27/make-ts.html", "title": "Make.ts", "date_modified": "2026-01-28T07:35:51.000Z" }, { "content_html": "Comments", "url": "https://allenai.org/blog/open-coding-agents", "title": "AI2: Open Coding Agents", "date_modified": "2026-01-27T17:17:54.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/aperture-private-alpha", "title": "A first look at Aperture by Tailscale (private alpha)", "date_modified": "2026-01-27T16:23:40.000Z" }, { "content_html": "Comments", "url": "https://9to5mac.com/2026/01/23/tiktok-is-officially-us-owned-for-american-users-heres-whats-changing/", "title": "TikTok is officially US-owned for American users, here's what's changing", "date_modified": "2026-01-25T04:37:58.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/NicerInPerson/status/2014989679796347375", "title": "Claude Code's new hidden feature: Swarms", "date_modified": "2026-01-24T14:35:47.000Z" }, { "content_html": "Comments", "url": "https://tuananh.net/2026/01/20/what-has-docker-become/", "title": "What has Docker become?", "date_modified": "2026-01-23T12:36:17.000Z" }, { "content_html": "Comments", "url": "https://lix.dev/blog/introducing-lix/", "title": "Lix – universal version control system for binary files", "date_modified": "2026-01-21T23:55:06.000Z" }, { "content_html": "Comments", "url": "https://cognition.ai/blog/devin-review", "title": "Devin Review: AI to Stop Slop", "date_modified": "2026-01-21T21:09:35.000Z" }, { "content_html": "Comments", "url": "https://github.com/trasta298/keifu", "title": "Keifu – A TUI for navigating commit graphs with color and clarity", "date_modified": "2026-01-17T00:32:24.000Z" }, { "content_html": "Comments", "url": "https://www.zettalane.com/blog/openzfs-summit-2025-mayanas-objbacker.html", "title": "Native ZFS VDEV for Object Storage (OpenZFS Summit)", "date_modified": "2026-01-14T18:49:37.000Z" }, { "content_html": "Comments", "url": "https://usetero.com/blog/the-question-your-observability-vendor-wont-answer", "title": "I built Vector. Now I'm answering the question your observability vendor won't", "date_modified": "2026-01-14T16:07:34.000Z" }, { "content_html": "Comments", "url": "https://starlink.com/support/article/58c9c8b7-474e-246f-7e3c-06db3221d34d", "title": "Roam 50GB is now Roam 100GB", "date_modified": "2026-01-14T16:03:11.000Z" }, { "content_html": "Comments", "url": "https://xlii.space/eng/i-hate-github-actions-with-passion/", "title": "I Hate GitHub Actions with Passion", "date_modified": "2026-01-14T10:53:13.000Z" }, { "content_html": "Comments", "url": "https://www.greptile.com/blog/github-ids", "title": "Every GitHub object has two IDs", "date_modified": "2026-01-13T15:52:33.000Z" }, { "content_html": "Comments", "url": "https://aicoding.leaflet.pub/3mcbiyal7jc2y", "title": "Provenance Is the New Version Control", "date_modified": "2026-01-13T03:26:23.000Z" }, { "content_html": "Comments", "url": "https://jakobemmerling.de/posts/fuse-is-all-you-need/", "title": "FUSE is All You Need – Giving agents access to anything via filesystems", "date_modified": "2026-01-11T21:12:45.000Z" }, { "content_html": "

Comments

", "url": "https://nilfs.sourceforge.io/en/index.html", "title": "Continuous Snapshotting Filesystem", "date_modified": "2026-01-10T12:15:20.000Z" }, { "content_html": "

Comments

", "url": "https://abhinavsarkar.net/posts/jj-usage/", "title": "How I use Jujutsu", "date_modified": "2026-01-10T11:55:32.000Z" }, { "content_html": "

Search engine optimization, or SEO, is a big business. While some SEO practices are useful, much of the day-to-day SEO wisdom you see online amounts to superstition. An increasingly popular approach geared toward LLMs called \"content chunking\" may fall into that category. In the latest installment of Google's Search Off the Record podcast, John Mueller and Danny Sullivan say that breaking content down into bite-sized chunks for LLMs like Gemini is a bad idea.

\n

You've probably seen websites engaging in content chunking and scratched your head, and for good reason—this content isn't made for you. The idea is that if you split information into smaller paragraphs and sections, it is more likely to be ingested and cited by generative AI bots like Gemini. So you end up with short paragraphs, sometimes with just one or two sentences, and lots of subheds formatted like questions one might ask a chatbot.

\n

According to Sullivan, this is a misconception, and Google doesn't use such signals to improve ranking. \"One of the things I keep seeing over and over in some of the advice and guidance and people are trying to figure out what do we do with the LLMs or whatever, is that turn your content into bite-sized chunks, because LLMs like things that are really bite size, right?\" said Sullivan. \"So... we don't want you to do that.\"

Read full article

\n

Comments

", "url": "https://arstechnica.com/google/2026/01/google-dont-make-bite-sized-content-for-llms-if-you-care-about-search-rank/", "title": "Google: Don’t make “bite-sized” content for LLMs if you care about search rank", "date_modified": "2026-01-09T20:27:34.000Z" }, { "content_html": "

Comments

", "url": "https://ploum.net/2026-01-05-unteaching_github.html", "title": "How GitHub monopoly is destroying the open source ecosystem", "date_modified": "2026-01-05T14:25:50.000Z" }, { "content_html": "Comments", "url": "https://github.com/huseyinbabal/taws", "title": "Show HN: Terminal UI for AWS", "date_modified": "2026-01-04T20:17:22.000Z" }, { "content_html": "Comments", "url": "https://github.com/quickemu-project/quickemu", "title": "Quickemu: Quickly create and run optimised Windows, macOS and Linux VMs", "date_modified": "2025-12-30T08:19:08.000Z" }, { "content_html": "Comments", "url": "https://rocketedge.com/2025/12/29/vibe-coding-for-ctos-the-real-cost-of-100-lines-of-code-ai-agents-vs-human-developers-without-losing-control/", "title": "Vibe Coding for CTOs: The Real Cost of 100 Lines of Code", "date_modified": "2025-12-29T02:33:49.000Z" }, { "content_html": "Comments", "url": "https://exe.dev/", "title": "Exe.dev/", "date_modified": "2025-12-26T23:42:46.000Z" }, { "content_html": "Comments", "url": "https://nesbitt.io/2025/12/24/package-managers-keep-using-git-as-a-database.html", "title": "Package managers keep using Git as a database, it never works out", "date_modified": "2025-12-26T12:46:36.000Z" }, { "content_html": "Comments", "url": "https://www.hobson.space/posts/nixcross/", "title": "Custom Cross Compiler with Nix", "date_modified": "2025-12-24T05:31:45.000Z" }, { "content_html": "Comments", "url": "https://willmcgugan.github.io/toad-released/", "title": "Toad is a unified experience for AI in the terminal", "date_modified": "2025-12-22T15:12:40.000Z" }, { "content_html": "Comments", "url": "https://loggingsucks.com/", "title": "Logging Sucks", "date_modified": "2025-12-21T18:09:52.000Z" }, { "content_html": "Comments", "url": "https://graphite.com/blog/graphite-joins-cursor", "title": "Cursor Acquires Graphite", "date_modified": "2025-12-19T16:07:42.000Z" }, { "content_html": "Comments", "url": "https://www.blacksmith.sh/blog/actions-pricing", "title": "The GitHub Actions control plane is no longer free", "date_modified": "2025-12-16T17:37:34.000Z" }, { "content_html": "Comments", "url": "https://resources.github.com/actions/2026-pricing-changes-for-github-actions/", "title": "Pricing Changes for GitHub Actions", "date_modified": "2025-12-16T17:12:02.000Z" }, { "content_html": "Comments", "url": "https://jyn.dev/what-is-a-build-system-anyway/", "title": "What is a build system, anyway?", "date_modified": "2025-12-13T19:58:32.000Z" }, { "content_html": "Comments", "url": "https://hatchet.run/blog/durable-execution", "title": "How to think about durable execution", "date_modified": "2025-12-12T15:49:16.000Z" }, { "content_html": "Comments", "url": "https://github.com/hashicorp/terraform-cdk", "title": "The future of Terraform CDK", "date_modified": "2025-12-10T19:14:03.000Z" }, { "content_html": "Comments", "url": "https://detail.dev/", "title": "Show HN: Detail, a Bug Finder", "date_modified": "2025-12-09T17:35:35.000Z" }, { "content_html": "Comments", "url": "https://dagger.io/", "title": "Dagger: Define software delivery workflows and dev environments", "date_modified": "2025-12-09T08:32:51.000Z" }, { "content_html": "Comments", "url": "https://nesbitt.io/2025/12/06/github-actions-package-manager.html", "title": "GitHub Actions Has a Package Manager, and It Might Be the Worst", "date_modified": "2025-12-08T08:15:32.000Z" }, { "content_html": "Comments", "url": "https://nesbitt.io/2025/12/06/github-actions-package-manager.html", "title": "GitHub Actions has a package manager, and it might be the worst", "date_modified": "2025-12-08T08:15:32.000Z" }, { "content_html": "Comments", "url": "https://github.com/observIQ/otel-distro-builder", "title": "OpenTelemetry Distribution Builder", "date_modified": "2025-12-06T23:41:18.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/5-december-2025-outage/", "title": "Cloudflare outage on December 5, 2025", "date_modified": "2025-12-05T15:35:43.000Z" }, { "content_html": "Comments", "url": "https://about.netflix.com/en/news/netflix-to-acquire-warner-bros", "title": "Netflix to Acquire Warner Bros. In an $82.7B Deal", "date_modified": "2025-12-05T12:21:19.000Z" }, { "content_html": "Comments", "url": "https://blog.ui.com/article/introducing-unifi-5g", "title": "UniFi 5G", "date_modified": "2025-12-05T07:06:38.000Z" }, { "content_html": "Use Supabase as a platform for your own business and tools", "url": "https://supabase.com/blog/introducing-supabase-for-platforms", "title": "Introducing Supabase for Platforms", "date_modified": "2025-12-05T07:00:00.000Z" }, { "content_html": "Comments", "url": "https://mitchellh.com/writing/ghostty-non-profit", "title": "Ghostty Is Now Non-Profit", "date_modified": "2025-12-03T18:40:06.000Z" }, { "content_html": "Comments", "url": "https://bun.com/blog/bun-joins-anthropic", "title": "Anthropic acquires Bun", "date_modified": "2025-12-02T18:05:44.000Z" }, { "content_html": "Comments", "url": "https://devblogs.microsoft.com/typescript/progress-on-typescript-7-december-2025/", "title": "Progress on TypeScript 7 – December 2025", "date_modified": "2025-12-02T17:37:06.000Z" }, { "content_html": "

Modern applications increasingly require complex and long-running coordination between services, such as multi-step payment processing, AI agent orchestration, or approval processes awaiting human decisions. Building these traditionally required significant effort to implement state management, handle failures, and integrate multiple infrastructure services.

\n

Starting today, you can use AWS Lambda durable functions to build reliable multi-step applications directly within the familiar AWS Lambda experience. Durable functions are regular Lambda functions with the same event handler and integrations you already know. You write sequential code in your preferred programming language, and durable functions track progress, automatically retry on failures, and suspend execution for up to one year at defined points, without paying for idle compute during waits.

\n

AWS Lambda durable functions use a checkpoint and replay mechanism, known as durable execution, to deliver these capabilities. After enabling a function for durable execution, you add the new open source durable execution SDK to your function code. You then use SDK primitives like “steps” to add automatic checkpointing and retries to your business logic and “waits” to efficiently suspend execution without compute charges. When execution terminates unexpectedly, Lambda resumes from the last checkpoint, replaying your event handler from the beginning while skipping completed operations.

\n

Getting started with AWS Lambda durable functions
Let me walk you through how to use durable functions.

\n

First, I create a new Lambda function in the console and select Author from scratch. In the Durable execution section, I select Enable. Note that, durable function setting can only be set during function creation and currently can’t be modified for existing Lambda functions.

\n

\n

After I create my Lambda durable function, I can get started with the provided code.

\n

\n

Lambda durable functions introduces two core primitives that handle state management and recovery:

\n \n

Additionally, Lambda durable functions provides other operations for more complex patterns: create_callback() creates a callback that you can use to await results for external events like API responses or human approvals, wait_for_condition() pauses until a specific condition is met like polling a REST API for process completion, and parallel() or map() operations for advanced concurrency use cases.

\n

Building a production-ready order processing workflow
Now let’s expand the default example to build a production-ready order processing workflow. This demonstrates how to use callbacks for external approvals, handle errors properly, and configure retry strategies. I keep the code intentionally concise to focus on these core concepts. In a full implementation, you could enhance the validation step with Amazon Bedrock to add AI-powered order analysis.

\n

Here’s how the order processing workflow works:

\n \n

Here’s the complete order processing workflow with step definitions and the main handler:

\n
import random\nfrom aws_durable_execution_sdk_python import (\n    DurableContext,\n    StepContext,\n    durable_execution,\n    durable_step,\n)\nfrom aws_durable_execution_sdk_python.config import (\n    Duration,\n    StepConfig,\n    CallbackConfig,\n)\nfrom aws_durable_execution_sdk_python.retries import (\n    RetryStrategyConfig,\n    create_retry_strategy,\n)\n\n\n@durable_step\ndef validate_order(step_context: StepContext, order_id: str) -> dict:\n    \"\"\"Validates order data using AI.\"\"\"\n    step_context.logger.info(f\"Validating order: {order_id}\")\n    # In production: calls Amazon Bedrock to validate order completeness and accuracy\n    return {\"order_id\": order_id, \"status\": \"validated\"}\n\n\n@durable_step\ndef send_for_approval(step_context: StepContext, callback_id: str, order_id: str) -> dict:\n    \"\"\"Sends order for approval using the provided callback token.\"\"\"\n    step_context.logger.info(f\"Sending order {order_id} for approval with callback_id: {callback_id}\")\n    \n    # In production: send callback_id to external approval system\n    # The external system will call Lambda SendDurableExecutionCallbackSuccess or\n    # SendDurableExecutionCallbackFailure APIs with this callback_id when approval is complete\n    \n    return {\n        \"order_id\": order_id,\n        \"callback_id\": callback_id,\n        \"status\": \"sent_for_approval\"\n    }\n\n\n@durable_step\ndef process_order(step_context: StepContext, order_id: str) -> dict:\n    \"\"\"Processes the order with retry logic for transient failures.\"\"\"\n    step_context.logger.info(f\"Processing order: {order_id}\")\n    # Simulate flaky API that sometimes fails\n    if random.random() > 0.4:\n        step_context.logger.info(\"Processing failed, will retry\")\n        raise Exception(\"Processing failed\")\n    return {\n        \"order_id\": order_id,\n        \"status\": \"processed\",\n        \"timestamp\": \"2025-11-27T10:00:00Z\",\n    }\n\n\n@durable_execution\ndef lambda_handler(event: dict, context: DurableContext) -> dict:\n    try:\n        order_id = event.get(\"order_id\")\n        \n        # Step 1: Validate the order\n        validated = context.step(validate_order(order_id))\n        if validated[\"status\"] != \"validated\":\n            raise Exception(\"Validation failed\")  # Terminal error - stops execution\n        context.logger.info(f\"Order validated: {validated}\")\n        \n        # Step 2: Create callback\n        callback = context.create_callback(\n            name=\"awaiting-approval\",\n            config=CallbackConfig(timeout=Duration.from_minutes(3))\n        )\n        context.logger.info(f\"Created callback with id: {callback.callback_id}\")\n        \n        # Step 3: Send for approval with the callback_id\n        approval_request = context.step(send_for_approval(callback.callback_id, order_id))\n        context.logger.info(f\"Approval request sent: {approval_request}\")\n        \n        # Step 4: Wait for the callback result\n        # This blocks until external system calls SendDurableExecutionCallbackSuccess or SendDurableExecutionCallbackFailure\n        approval_result = callback.result()\n        context.logger.info(f\"Approval received: {approval_result}\")\n        \n        # Step 5: Process the order with custom retry strategy\n        retry_config = RetryStrategyConfig(max_attempts=3, backoff_rate=2.0)\n        processed = context.step(\n            process_order(order_id),\n            config=StepConfig(retry_strategy=create_retry_strategy(retry_config)),\n        )\n        if processed[\"status\"] != \"processed\":\n            raise Exception(\"Processing failed\")  # Terminal error\n        \n        context.logger.info(f\"Order successfully processed: {processed}\")\n        return processed\n        \n    except Exception as error:\n        context.logger.error(f\"Error processing order: {error}\")\n        raise error  # Re-raise to fail the execution\n
\n

This code demonstrates several important concepts:

\n \n

Now I create a test event with order_id and invoke the function asynchronously to start the order workflow. I navigate to the Test tab and fill in the optional Durable execution name to identify this execution. Note that, durable functions provides built-in idempotency. If I invoke the function twice with the same execution name, the second invocation returns the existing execution result instead of creating a duplicate.

\n

\n

I can monitor the execution by navigating to the Durable executions tab in the Lambda console:

\n

\n

Here I can see each step’s status and timing. The execution shows CallbackStarted followed by InvocationCompleted, which indicates the function has terminated and execution is suspended to avoid idle charges while waiting for the approval callback.

\n

\n

I can now complete the callback directly from the console by choosing Send success or Send failure, or programmatically using the Lambda API.

\n

\n

I choose Send success.

\n

\n

After the callback completes, the execution resumes and processes the order. If the process_order step fails due to the simulated flaky API, it automatically retries based on the configured strategy. Once all retries succeed, the execution completes successfully.

\n

\n

Monitoring executions with Amazon EventBridge
You can also monitor durable function executions using Amazon EventBridge. Lambda automatically sends execution status change events to the default event bus, allowing you to build downstream workflows, send notifications, or integrate with other AWS services.

\n

To receive these events, create an EventBridge rule on the default event bus with this pattern:

\n
{\n  \"source\": [\"aws.lambda\"],\n  \"detail-type\": [\"Durable Execution Status Change\"]\n}\n
\n

Things to know
Here are key points to note:

\n \n

Get started with AWS Lambda durable functions by visiting the AWS Lambda console. To learn more, refer to AWS Lambda durable functions documentation page.

\n

Happy building!

\n

— Donnie

", "url": "https://aws.amazon.com/blogs/aws/build-multi-step-applications-and-ai-workflows-with-aws-lambda-durable-functions/", "title": "Build multi-step applications and AI workflows with AWS Lambda durable functions", "date_modified": "2025-12-02T16:12:19.000Z" }, { "content_html": "

Today, we’re announcing the public preview of AWS DevOps Agent, a frontier agent that helps you respond to incidents, identify root causes, and prevent future issues through systematic analysis of past incidents and operational patterns.

\n

Frontier agents represent a new class of AI agents that are autonomous, massively scalable, and work for hours or days without constant intervention.

\n

When production incidents occur, on-call engineers face significant pressure to quickly identify root causes while managing stakeholder communications. They must analyze data across multiple monitoring tools, review recent deployments, and coordinate response teams. After service restoration, teams often lack bandwidth to transform incident learnings into systematic improvements.

\n

AWS DevOps Agent is your always-on, autonomous on-call engineer. When issues arise, it automatically correlates data across your operational toolchain, from metrics and logs to recent code deployments in GitHub or GitLab. It identifies probable root causes and recommends targeted mitigations, helping reduce mean time to resolution. The agent also manages incident coordination, using Slack channels for stakeholder updates and maintaining detailed investigation timelines.

\n

To get started, you connect AWS DevOps Agent to your existing tools through the AWS Management Console. The agent works with popular services such as Amazon CloudWatch, Datadog, Dynatrace, New Relic, and Splunk for observability data, while integrating with GitHub Actions and GitLab CI/CD to track deployments and their impact on your cloud resources. Through the bring your own (BYO) Model Context Protocol (MCP) server capability, you can also integrate additional tools such as your organization’s custom tools, specialized platforms or open source observability solutions, such as Grafana and Prometheus into your investigations.

\n

The agent acts as a virtual team member and can be configured to automatically respond to incidents from your ticketing systems. It includes built-in support for ServiceNow, and through configurable webhooks, can respond to events from other incident management tools like PagerDuty. As investigations progress, the agent updates tickets and relevant Slack channels with its findings. All of this is powered by an intelligent application topology the agent builds—a comprehensive map of your system components and their interactions, including deployment history that helps identify potential deployment-related causes during investigations.

\n

Let me show you how it works
To show you how it works, I deployed a straigthforward AWS Lambda function that intentionally generates errors when invoked. I deployed it in an AWS CloudFormation stack.

\n

Step 1: Create an Agent Space

\n

An Agent Space defines the scope of what AWS DevOps Agent can access as it performs tasks.

\n

You can organize Agent Spaces based on your operational model. Some teams align an Agent Space with a single application, others create one per on-call team managing multiple services, and some organizations use a centralized approach. For this demonstration, I’ll show you how to create an Agent Space for a single application. This setup helps isolate investigations and resources for that specific application, making it easier to track and analyze incidents within its context.

\n

In the AWS DevOps Agent section of the AWS Management Console, I select Create Agent Space, enter a name for this space and create the AWS Identity and Access Management (IAM) roles it uses to introspect AWS resources in my or others’ AWS accounts.

\n

\"AWSFor this demo, I choose to enable the AWS DevOps Agent web app; more about this later. This can be done at a later stage.

\n

When ready, I choose Create.

\n

\"AWSAfter it has been created, I choose the Topology tab.

\n

This view shows the key resources, entities, and relationships AWS DevOps Agent has selected as a foundation for performing its tasks efficiently. It doesn’t represent everything AWS DevOps Agent can access or see, only what the Agent considers most relevant right now. By default, the Topology includes the AWS resources that are contained in my account. As your agent completes more tasks, it will discover and add new resources to this list.

\n

\"AWS

\n

Step 2: Configure the AWS DevOps web app for the operators

\n

The AWS DevOps Agent web app provides a web interface for on-call engineers to manually trigger investigations, view investigation details including relevant topology elements, steer investigations, and ask questions about an investigation.

\n

I can access the web app directly from my Agent Space in the AWS console by choosing the Operator access link. Alternatively, I can use AWS IAM Identity Center to configure user access for my team. IAM Identity Center lets me manage users and groups directly or connect to an identity provider (IdP), providing a centralized way to control who can access the AWS DevOps Agent web app.

\n

\"AWS

\n

At this stage, I have an Agent Space all set up to focus investigations and resources for this specific application, and I’ve enabled the DevOps team to initiate investigations using the web app.

\n

Now that the one-time setup for this application is done, I start invoking the faulty Lambda function. It generates errors at each invocation. The CloudWatch alarm associated with the Lambda errors count turns on to ALARM state. In real life, you might receive an alert from external services, such as ServiceNow. You can configure AWS DevOps Agent to automatically start investigations when receiving such alerts.

\n

For this demo, I manually start the investigation by selecting Start Investigation.

\n

You can also choose from several preconfigured starting points to quickly begin your investigation: Latest alarm to investigate your most recent triggered alarm and analyze the underlying metrics and logs to determine the root cause, High CPU usage to investigate high CPU utilization metrics across your compute resources and identify which processes or services are consuming excessive resources, or Error rate spike to investigate the recent increase in application error rates by analyzing metrics, application logs, and identifying the source of failures.

\n

\"AWS

\n

I enter some information, such as Investigation details, Investigation starting point, the Date and time of the incident, the AWS Account ID for the incident.

\n

\"-

\n

In the AWS DevOps Agent web app, you can watch the investigation unfold in real time. The agent identifies the application stack. It correlates metrics from CloudWatch, examines logs from CloudWatch Logs or external sources, such as Splunk, reviews recent code changes from GitHub, and analyzes traces from AWS X-Ray.

\n

\"-

\n

It identifies the error patterns and provides a detailed investigation summary. In the context of this demo, the investigation reveals that these are intentional test exceptions, shows the timeline of function invocations leading to the alarm, and even suggests monitoring improvements for error handling.

\n

The agent uses a dedicated incident channel in Slack, notifies on-call teams if needed, and provides real-time status updates to stakeholders. Through the investigation chat interface, you can interact directly with the agent by asking clarifying questions such as “which logs did you analyze?” or steering the investigation by providing additional context, such as “focus on these specific log groups and rerun your analysis.” If you need expert assistance, you can create an AWS Support case with a single click, automatically populating it with the agent’s findings, and engage with AWS Support experts directly through the investigation chat window.

\n

For this demo, the AWS DevOps Agent correctly identified manual activities in the Lambda console to invoke a function that intentionally triggers errors \"😇\".

\n

\"-

\n

Beyond incident response, AWS DevOps Agent analyzes my recent incidents to identify high-impact improvements that prevent future issues.

\n

During active incidents, the agent offers immediate mitigation plans through its incident mitigations tab to help restore service quickly. Mitigation plans consist of specs that provide detailed implementation guidance for developers and agentic development tools like Kiro.

\n

For longer-term resilience, it identifies potential enhancements by examining gaps in observability, infrastructure configurations, and deployment pipeline. My straightforward demo that triggered intentional errors was not enough to generate relevant recommendations though.

\n

\"AWS

\n

For example, it might detect that a critical service lacks multi-AZ deployment and comprehensive monitoring. The agent then creates detailed recommendations with implementation guidance, considering factors like operational impact and implementation complexity. In an upcoming quick follow-up release, the agent will expand its analysis to include code bugs and testing coverage improvements.

\n

Availability
You can try AWS DevOps Agent today in the US East (N. Virginia) Region. Although the agent itself runs in US East (N. Virginia) (us-east-1), it can monitor applications deployed in any Region, across multiple AWS accounts.

\n

During the preview period, you can use AWS DevOps Agent at no charge, but there will be a limit on the number of agent task hours per month.

\n

As someone who has spent countless nights debugging production issues, I’m particularly excited about how AWS DevOps Agent combines deep operational insights with practical, actionable recommendations. The service helps teams move from reactive firefighting to proactive system improvement.

\n

To learn more and sign up for the preview, visit AWS DevOps Agent. I look forward to hearing how AWS DevOps Agent helps improve your operational efficiency.

\n

— seb

", "url": "https://aws.amazon.com/blogs/aws/aws-devops-agent-helps-you-accelerate-incident-response-and-improve-system-reliability-preview/", "title": "AWS DevOps Agent helps you accelerate incident response and improve system reliability (preview)", "date_modified": "2025-12-02T16:05:42.000Z" }, { "content_html": "Comments", "url": "https://chrisebert.net/comparing-aws-lambda-arm64-vs-x86_64-performance-across-multiple-runtimes-in-late-2025/", "title": "Comparing AWS Lambda ARM64 vs. x86_64 Performance Across Runtimes in Late 2025", "date_modified": "2025-12-02T09:11:41.000Z" }, { "content_html": "

Comments

", "url": "https://nixiesearch.substack.com/p/benchmarking-read-latency-of-aws", "title": "Benchmarking read latency of AWS S3, S3 Express, EBS and Instance store", "date_modified": "2025-12-01T21:03:23.000Z" }, { "content_html": "Comments", "url": "https://github.com/coder/ghostty-web", "title": "Ghostty compiled to WASM with xterm.js API compatibility", "date_modified": "2025-12-01T18:17:02.000Z" }, { "content_html": "Comments", "url": "https://dineshpandiyan.com/blog/stacked-diffs-with-rebase-onto/", "title": "Stacked Diffs with git rebase —onto", "date_modified": "2025-12-01T04:47:30.000Z" }, { "content_html": "Comments", "url": "https://www.humanlayer.dev/blog/writing-a-good-claude-md", "title": "Writing a good Claude.md", "date_modified": "2025-11-30T17:56:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/BinSquare/ERA", "title": "Show HN: Era – Open-source local sandbox for AI agents", "date_modified": "2025-11-27T05:28:50.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=46045987", "title": "Launch HN: Onyx (YC W24) – Open-source chat UI", "date_modified": "2025-11-25T14:20:30.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/engineering/advanced-tool-use", "title": "Claude Advanced Tool Use", "date_modified": "2025-11-24T19:21:35.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/claude-opus-4-5", "title": "Claude Opus 4.5", "date_modified": "2025-11-24T18:53:05.000Z" }, { "content_html": "Comments", "url": "https://www.cloudflarestatus.com/incidents/8gmgl950y3h7", "title": "Cloudflare Global Network experiencing issues", "date_modified": "2025-11-18T11:48:56.000Z" }, { "content_html": "Comments", "url": "https://www.cloudflarestatus.com/?t=1", "title": "Cloudflare, X, More are down", "date_modified": "2025-11-18T11:35:10.000Z" }, { "content_html": "Comments", "url": "https://excalidraw.com/#json=kMtNOJfH_UUOzBqt7WXx9,cyuXonQjb-Kor72f0F5YXg", "title": "Show HN: A visual guide to learning Jujutsu (JJ)", "date_modified": "2025-11-15T00:32:04.000Z" }, { "content_html": "Comments", "url": "https://www.claude.com/blog/structured-outputs-on-the-claude-developer-platform", "title": "Structured outputs on the Claude Developer Platform", "date_modified": "2025-11-14T19:04:23.000Z" }, { "content_html": "

Today, we’re announcing the general availability of provisioned mode for AWS Lambda with Amazon Simple Queue Service (Amazon SQS) Event Source Mapping (ESM), a new feature that customers can use to optimize the throughput of their event-driven applications by configuring dedicated polling resources. Using this new capability, which provides 3x faster scaling, and 16x higher concurrency, you can process events with lower latency, handle sudden traffic spikes more effectively, and maintain precise control over your event processing resources.

\n

Modern applications increasingly rely on event-driven architectures where services communicate through events and messages. Amazon SQS is commonly used as an event source for Lambda functions, so developers can build loosely coupled, scalable applications. Although the SQS ESM automatically handles queue polling and function invocation, customers with stringent performance requirements have asked for more control over the polling behavior to handle spiky traffic patterns and maintain low processing latency.

\n

Provisioned mode for SQS ESM addresses these needs by introducing event pollers, which are dedicated resources that remain ready to handle expected traffic patterns. These event pollers can auto scale up to 1000 per concurrent executions per minute, more than three times faster than before to handle sudden spikes in event traffic and provide up to 20,000 concurrency–16 times higher capacity to process millions of events with Lambda functions. This enhanced scaling behavior helps customers maintain predictable low latency even during traffic surges.

\n

Enterprises across various industries, from financial services to gaming companies, are using AWS Lambda with Amazon SQS to process real-time events for their mission-critical applications. These organizations, which include some of the largest online gaming platforms and financial institutions, require consistent subsecond processing times for their event-driven workloads, particularly during periods of peak usage. Provisioned mode for SQS ESM is a capability you can use to meet your stringent performance requirements while maintaining cost controls.

\n

Enhanced control and performance

\n

With provisioned mode, you can configure both minimum and maximum numbers of event pollers for your SQS ESM. Each event poller represents a unit of compute that handles queue polling, event batching, and filtering before invoking Lambda functions. Each event poller can handle up to 1 MB/sec of throughput, up to 10 concurrent invokes, or up to 10 SQS polling API calls per second. By setting a minimum number of event pollers, you enable your application to maintain a baseline processing capacity that can immediately handle sudden traffic increases. We recommend that you set the minimum event pollers required to handle your known peak workload requirements. The optional maximum setting helps prevent overloading downstream systems by limiting the total processing throughput.

\n

The new mode delivers significant improvements in how your event-driven applications handle varying workloads. When traffic increases, your ESM detects the growing backlog within seconds and dynamically scales event pollers between your configured minimum and maximum values three times faster than before. This enhanced scaling capability is complemented by a substantial increase in processing capacity, with support for up to 2 GBps of aggregate traffic, and up to 20K concurrent requests—16x higher than previously possible. By maintaining a minimum number of ready-to-use event pollers, your application achieves predictable performance, handling sudden traffic spikes without the delay typically associated with scaling up resources. During low traffic periods, your ESM automatically scales down to your configured minimum number of event pollers, which means you can optimize costs while maintaining responsiveness.

\n

Let’s try it out

\n

Enabling provisioned mode is straightforward in the AWS Management Console. You need to already have an SQS queue configured and a Lambda function. To get started, in the Configuration tab for your Lambda function, choose Triggers, then Add trigger. This will bring up a user interface where you can configure your trigger. Choose SQS from the dropdown menu for source and then select the SQS queue you want to use.

\n

Under Event poller configuration, you will now see a new option called Provisioned mode. Select Configure to reveal settings for Minimum event pollers and Maximum event pollers, each with defaults and minimum and maximum values displayed.

\n

\"Configuration

\n

After you have configured Provisioned mode, you can save your trigger. If you need to make changes later, you can find the current configuration under the Triggers tab in the AWS Lambda configuration section, and you can modify your current settings there.

\n

\"SQS

\n

Monitoring and observability

\n

You can monitor your provisioned mode usage through Amazon CloudWatch metrics. The ProvisionedPollers metric shows the number of active event pollers processing events in one-minute windows.

\n

Now available

\n

Provisioned mode for Lambda SQS ESM is available today in all commercial AWS Regions. You can start using this feature through the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDKs. Pricing is based on the number of event pollers provisioned and the duration they’re provisioned for, measured in Event Poller Units (EPUs). Each EPU supports up to 1 MB per second throughput capacity per event poller, with minimum 2 event pollers per ESM. See the AWS pricing page for more information on EPU charges.

\n

To learn more about provisioned mode for SQS ESM, visit the AWS Lambda documentation. Start building more responsive event-driven applications today with enhanced control over your event processing resources.

", "url": "https://aws.amazon.com/blogs/aws/aws-lambda-enhances-sqs-processing-with-new-provisioned-mode-3x-faster-scaling-16x-higher-capacity/", "title": "AWS Lambda enhances event processing with provisioned mode for SQS event-source mapping", "date_modified": "2025-11-14T17:45:04.000Z" }, { "content_html": "Comments", "url": "https://seed.line.me/index_en.html", "title": "Seed. LINE's Custom Typeface", "date_modified": "2025-11-13T09:36:51.000Z" }, { "content_html": "

Google and the cloud-native community have consistently strengthened Kubernetes to support modern applications. At KubeCon EU 2025 earlier this year, we announced a series of enhancements to Kubernetes to better support AI inference. Today, at KubeCon NA 2025, we’re focused on making Kubernetes the most open and scalable platform for AI agents, with the introduction of Agent Sandbox.

\n

Consider the challenge that AI agents represent. AI agents help applications go from answering simple queries to performing complex, multi-step tasks to achieve the users objective. Provided a request like “visualize last quarters sales data”, the agent has to use one tool to query the data and another to process that data into a graph and return to the user.  Where traditional software is predictable, AI agents can make their own decisions about when and how to use tools at their disposal to achieve a user's objective, including generating code, using computer terminals and even browsers.

\n

Without strong security and operational guardrails, orchestrating powerful, non-deterministic agents can introduce significant risks. Providing kernel-level isolation for agents that execute code and commands is non-negotiable. AI and agent-based workloads also have additional infrastructure needs compared to traditional applications. Most notably, they need to orchestrate thousands of sandboxes as ephemeral environments, rapidly creating and deleting them as needed while ensuring they have limited network access.  

\n

With its maturity, security, and scalability, we believe Kubernetes provides the most suitable foundation for running AI agents. Yet it still needs to evolve to meet the needs of agent code execution and computer use scenarios. Agent Sandbox is a powerful first step in that direction. 

\n

Strong isolation at scale

\n

Agentic code execution and computer use require an isolated sandbox to be provisioned for each task. Further, users expect infrastructure to keep pace even as thousands of sandboxes are scheduled in parallel. 

\n

At its core, Agent Sandbox is a new Kubernetes primitive built with the Kubernetes community that’s designed specifically for agent code execution and computer use, delivering the performance and scale needed for the next generation of agentic AI workloads. Foundationally built on gVisor with additional support for Kata Containers for runtime isolation, Agent Sandbox provides a secure boundary to reduce the risk of vulnerabilities that could lead to data loss, exfiltration or damage to production systems. We’re continuing our commitment to open source, building Agent Sandbox as a Cloud Native Computing Foundation (CNCF) project in the Kubernetes community.

\n
\n\n\n\n\n\n\n \n
\n
\n \n\n
\n\n \n \n \n \"1\"\n \n \n \n
\n\n \n
\n
\n \n\n\n\n\n
\n

Enhanced performance on GKE

\n

At the same time, you need to optimize performance as you scale your agents to deliver the best agent user-experience at the lowest cost. When you use Agent Sandbox on Google Kubernetes Engine (GKE), you can leverage managed gVisor in GKE Sandbox and the container-optimized compute platform to horizontally scale your sandboxes faster. Agent Sandbox also enables low-latency sandbox execution by enabling administrators to configure pre-warmed pools of sandboxes. With this feature, Agent Sandbox delivers sub-second latency for fully isolated agent workloads, up to a 90% improvement over cold starts.

\n

The same isolation property that makes a sandbox safe, makes it more susceptible to compute underutilization. Reinitializing each sandbox environment with a script can be brittle and slow, and idle sandboxes often waste valuable compute cycles. In a perfect world, you could take a snapshot of running sandbox environments to start them from a specific state.

\n

Pod Snapshots is a new, GKE-exclusive feature that enables full checkpoint and restore of running pods. Pod Snapshots drastically reduces startup latency of agent and AI workloads. When combined with Agent Sandbox, Pod Snapshots lets teams provision sandbox environments from snapshots, so they can start up in seconds. GKE Pod Snapshots supports snapshot and restore of both CPU- and GPU-based workloads, bringing pod start times from minutes down to seconds. With Pod Snapshots, any idle sandbox can be snapshotted and suspended, saving significant compute cycles with little to no disruption for end-users.

\n
\n\n\n\n\n\n\n \n
\n
\n \n\n
\n\n \n \n \n \"2\"\n \n \n \n
\n\n \n
\n
\n \n\n\n\n\n
\n

Built for AI engineers

\n

Teams building today’s agentic AI or reinforcement learning (RL) systems should not have to be infrastructure experts. We built Agent Sandbox with AI engineers in mind, designing an API and Python SDK that lets them manage the lifecycle of their sandboxes, without worrying about the underlying infrastructure. 

\n
\n
code_block
\n
<ListValue: [StructValue([('code', 'from agentic_sandbox import Sandbox\\r\\n\\r\\n# The SDK abstracts all YAML into a simple context manager \\r\\nwith Sandbox(template_name=\"python3-template\",namespace=\"ai-agents\") as sandbox:\\r\\n\\r\\n # Execute a command inside the sandbox\\r\\n result = sandbox.run(\"print(\\'Hello from inside the sandbox!\\')\")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fe6dcdaa070>)])]>
\n
\n

This separation of concern enables both an AI developer-friendly experience and the operational control and extensibility that Kubernetes administrators and operators expect.

\n

Get started today

\n

Agentic AI represents a profound shift for software development and infrastructure teams. Agent Sandbox and GKE can help  deliver the isolation and performance your agents need. Agent Sandbox is available in open source and can be deployed on GKE today. GKE Pod Snapshots is available in limited preview and will be available to all GKE customers later this year. To get started, check out the Agent Sandbox documentation  and quick start. We are excited to see what you build!

", "url": "https://cloud.google.com/blog/products/containers-kubernetes/agentic-ai-on-kubernetes-and-gke/", "title": "Introducing Agent Sandbox: Strong guardrails for agentic AI on Kubernetes and GKE", "date_modified": "2025-11-11T12:00:00.000Z" }, { "content_html": "Comments", "url": "https://maxdeviant.com/posts/2025/head-in-the-zed-cloud/", "title": "Head in the Zed Cloud", "date_modified": "2025-11-10T14:23:17.000Z" }, { "content_html": "

Comments

", "url": "https://maxdeviant.com/posts/2025/head-in-the-zed-cloud/", "title": "Head in the Zed Cloud", "date_modified": "2025-11-10T14:22:06.000Z" }, { "content_html": "Comments", "url": "https://www.thedailybeast.com/air-traffic-controllers-start-resigning-as-shutdown-bites/", "title": "US Air Traffic Controllers Start Resigning as Shutdown Bites", "date_modified": "2025-11-08T23:06:10.000Z" }, { "content_html": "Comments", "url": "https://github.com/openpcc/openpcc", "title": "Open Source Implementation of Apple's Private Compute Cloud", "date_modified": "2025-11-05T15:52:27.000Z" }, { "content_html": "Comments", "url": "https://cognition.ai/blog/codemaps", "title": "Codemaps: Understand Code, Before You Vibe It", "date_modified": "2025-11-04T17:47:09.000Z" }, { "content_html": "Comments", "url": "https://blog.yakkomajuri.com/blog/python-to-node", "title": "Why We Migrated from Python to Node.js", "date_modified": "2025-11-03T16:35:44.000Z" }, { "content_html": "Comments", "url": "https://lil.law.harvard.edu/blog/2025/10/24/rethinking-data-discovery-for-libraries-and-digital-humanities/", "title": "Use DuckDB-WASM to query TB of data in browser", "date_modified": "2025-10-31T17:37:15.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=v9Ob5yPpC0A", "title": "Jujutsu at Google [video]", "date_modified": "2025-10-30T13:00:58.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/peer-relays-beta", "title": "Tailscale Peer Relays", "date_modified": "2025-10-29T16:21:36.000Z" }, { "content_html": "Comments", "url": "https://www.tigerdata.com/blog/fluid-storage-forkable-ephemeral-durable-infrastructure-age-of-agents", "title": "Replacing EBS and Rethinking Postgres Storage from First Principles", "date_modified": "2025-10-29T15:49:18.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/services-beta", "title": "Tailscale Services", "date_modified": "2025-10-28T12:19:38.000Z" }, { "content_html": "Comments", "url": "https://www.iroh.computer/blog/iroh-blobs-0-95-new-features", "title": "Iroh-Blobs", "date_modified": "2025-10-27T23:28:13.000Z" }, { "content_html": "Comments", "url": "https://sealos.io/blog/reduce-container-image-size-case-study", "title": "We reduced a container image from 800GB to 2GB", "date_modified": "2025-10-27T10:15:24.000Z" }, { "content_html": "Comments", "url": "https://www.npmjs.com/package/vite-plugin-use-golang", "title": "Show HN: Write Go code in JavaScript files", "date_modified": "2025-10-27T05:36:13.000Z" }, { "content_html": "Comments", "url": "https://maurycyz.com/misc/easy_git/", "title": "You Already Have a Git Server", "date_modified": "2025-10-26T10:53:37.000Z" }, { "content_html": "Comments", "url": "https://tsdown.dev/", "title": "Tsdown – The Elegant Bundler for Libraries", "date_modified": "2025-10-26T03:19:08.000Z" }, { "content_html": "Comments", "url": "https://ducklake.select/2025/10/24/frozen-ducklake/", "title": "Frozen DuckLakes for Multi-User, Serverless Data Access", "date_modified": "2025-10-25T10:57:51.000Z" }, { "content_html": "Comments", "url": "https://www.seangoedecke.com/good-code-reviews/", "title": "Mistakes I see engineers making in their code reviews", "date_modified": "2025-10-25T04:42:09.000Z" }, { "content_html": "Comments", "url": "https://steveklabnik.com/writing/i-see-a-future-in-jj/", "title": "I see a future in jj", "date_modified": "2025-10-22T17:21:54.000Z" }, { "content_html": "

Comments

", "url": "https://judojj.com/", "title": "Judo (Jujutsu GUI)", "date_modified": "2025-10-21T00:14:55.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/claude-code-on-the-web", "title": "Claude Code on the Web", "date_modified": "2025-10-20T18:12:23.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/claude-code-on-the-web", "title": "Claude Code on the web", "date_modified": "2025-10-20T18:12:23.000Z" }, { "content_html": "Comments", "url": "https://www.xtxmarkets.com/tech/2025-ternfs/#posix-shaped", "title": "TernFS – an exabyte scale, multi-region distributed filesystem", "date_modified": "2025-10-20T17:36:16.000Z" }, { "content_html": "Comments", "url": "https://judojj.com", "title": "Show HN: I created a cross-platform GUI for the JJ VCS (Git compatible)", "date_modified": "2025-10-20T15:35:19.000Z" }, { "content_html": "Comments", "url": "https://old.reddit.com/r/aws/comments/1obd3lx/dynamodb_down_useast1/", "title": "Major AWS Outage Happening", "date_modified": "2025-10-20T07:11:06.000Z" }, { "content_html": "Comments", "url": "https://elixir-lang.org/blog/2025/10/16/elixir-v1-19-0-released/", "title": "Elixir 1.19", "date_modified": "2025-10-16T07:31:25.000Z" }, { "content_html": "

 

\n

As someone that used to work at Sun Microsystems, where ZFS was invented, I’ve always loved working with storage systems that offer instant volume copies for my development and testing needs.

\n

Today, I’m excited to share that AWS is bringing similar capabilities to Amazon Elastic Block Store (Amazon EBS) with the launch of Amazon EBS Volume Clones, a new capability that lets you create instant point-in-time copies of your EBS volumes within the same Availability Zone.

\n

Many customers need to create copies of their production data to support development and testing activities in a separate nonproduction environment. Until now, this process required taking an EBS snapshot (stored in Amazon Simple Storage Service (Amazon S3)) and then creating a new volume from that snapshot. Although this approach works, the process creates operational overhead due to multiple steps.

\n

With Amazon EBS Volume Clones, you can now create copies of your EBS volumes with a single API call or console click. The copied volumes are available within seconds and provide immediate access to your data with single-digit millisecond latency. This makes Volume Clones particularly useful for quickly setting up test environments with production data or creating temporary copies of databases for development purposes.

\n

Let me show you how Volume Clones works
For this post, I created a small Amazon Elastic Compute Cloud (Amazon EC2) instance, with an attached volume. I created a file on the root file system with the command echo \"Hello CopyVolumes\" > hello.txt.

\n

To initiate the copy, I open a browser on the AWS Management Console and I navigate to EC2, Elastic Block Store, Volumes. I select the volume I want to copy.

\n

Note that, at the time of publication of this post, only encrypted volumes can be copied.

\n

On the Actions menu, I choose the Copy Volume option.

\n

\"Copy

\n

Next, I choose the details of the target volume. I can change the Volume type and adjust the Size, IOPS, and Throughput parameters. I choose Copy volume to start the Volume Clone operation.

\n

\"Copy

\n

The copied volume enters the Creating state and becomes available within seconds. I can then attach it to an EC2 instance and start using it immediately.

\n

Data blocks are copied from the source volume and written to the volume copy in the background. The volume remains in the Initializing state until the process is complete. I can monitor its progress with the describe-volume-status API. The initializing operation doesn’t affect the performance of the source volume. I can continue using it normally during the copy process.

\n

I love that the copied volume is available immediately. I don’t need to wait for its initialization to complete. During the initialization phase, my copied volume delivers performance based on the lowest of: a baseline of 3,000 IOPS and 125 MiB/s, the source volume’s provisioned performance, or the copied volume’s provisioned performance.

\n

After initialization is completed, the copied volume becomes fully independent of the source volume and delivers its full provisioned performance.

\n

\"CopyAlternatively, I can use the AWS Command Line Interface (AWS CLI) to initiate the copy:

\n
aws ec2 copy-volumes                          \\\n     --source-volume-id vol-1234567890abcdef0 \\\n     --size 500                               \\\n     --volume-type gp3
\n

After the volume copy is created, I attach it to my EC2 instance and mount it. I can check the file I created at start is present.

\n

First, I attach the volume from my laptop, using the attach-volume command:

\n
aws ec2 attach-volume \\\n         --volume-id 'vol-09b700e3a23a9b4ad' \\\n         --instance-id 'i-079e6504ad25b029e'   \\\n         --device '/dev/sdb'
\n

Then, I connect to the instance, and I type these commands:

\n
$ sudo lsblk -f\nNAME          FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS\nnvme0n1                                                                              \n├─nvme0n1p1   xfs          /     49e26d9d-0a9d-4667-b93e-a23d1de8eacd    6.2G    22% /\n└─nvme0n1p128 vfat   FAT16       3105-2F44                               8.6M    14% /boot/efi\nnvme1n1                                                                              \n├─nvme1n1p1   xfs          /     49e26d9d-0a9d-4667-b93e-a23d1de8eacd                \n└─nvme1n1p128 vfat   FAT16       3105-2F44     \n\n$ sudo mount -t xfs /dev/nvme1n1p1 /data\n\n$ df -h\nFilesystem        Size  Used Avail Use% Mounted on\ndevtmpfs          4.0M     0  4.0M   0% /dev\ntmpfs             924M     0  924M   0% /dev/shm\ntmpfs             370M  476K  369M   1% /run\n/dev/nvme0n1p1    8.0G  1.8G  6.2G  22% /\ntmpfs             924M     0  924M   0% /tmp\n/dev/nvme0n1p128   10M  1.4M  8.7M  14% /boot/efi\ntmpfs             185M     0  185M   0% /run/user/1000\n/dev/nvme1n1p1    8.0G  1.8G  6.2G  22% /data\n\n$ cat /data/home/ec2-user/hello.txt \nHello CopyVolumes
\n

Things to know
Volume Clones creates copies within the same Availability Zone as your source volume. You can create copies from encrypted volumes only, and the size of your copy must be equal to or greater than the source volume.

\n

Volume Clones creates crash-consistent copies of your volumes, exactly like snapshots. For application consistency, you need to pause application I/O operations before creating the copy. For example, with PostgreSQL databases, you can use the pg_start_backup() and pg_stop_backup() functions to pause writes and create a consistent copy. At the operating system level on Linux with XFS, you can use the xfs_freeze command to temporarily suspend and resume access to the file system and ensure all cached updates are written to disk.

\n

Although Volume Clones creates point-in-time copies, it complements rather than replaces EBS snapshots for backup purposes. EBS snapshots remain the recommended solution for data backup and protection against AZ-level and volume failures. Snapshots provide incremental backups to Amazon S3 with 11 nines of durability, compared to Volume Clones which maintains EBS volume durability (99.999% for io2, 99.9% for other volume types). Consider using Volume Clones specifically for test and development environment scenarios where you need instant access to volume copies.

\n

Copied volumes exist independently of their source volumes and continue to incur standard EBS volume charges until you delete them. To manage costs effectively, implement governance rules to identify and remove copied volumes that are no longer needed for your development or testing activities.

\n

Pricing and availability
Volume Clones supports all EBS volume types and works with volumes in the same AWS account and Availability Zone. This new capability is available in all AWS commercial Regions, selected Local Zones, and in the AWS GovCloud (US).

\n

For pricing, you’re charged a one-time fee per GiB of data on the source volume at initiation and standard EBS pricing for the new volume.

\n

I find Volume Clones particularly valuable for database workloads and continuous integration (CI) scenarios. For instance, you can quickly create a copy of your production database for testing new features or troubleshooting issues without impacting your production environment or waiting for data to hydrate from Amazon S3.

\n

To get started with Amazon EBS Volume Clones, visit the Amazon EBS section on the console or check out the EBS documentation. I look forward to hearing how you use this capability to improve your development workflows.

\n— seb", "url": "https://aws.amazon.com/blogs/aws/introducing-amazon-ebs-volume-clones-create-instant-copies-of-your-ebs-volumes/", "title": "Introducing Amazon EBS Volume Clones: Create instant copies of your EBS volumes", "date_modified": "2025-10-14T21:35:05.000Z" }, { "content_html": "Comments", "url": "https://mtlynch.io/notes/hold-off-on-litestream-0.5.0/", "title": "Hold Off on Litestream 0.5.0", "date_modified": "2025-10-14T16:10:27.000Z" }, { "content_html": "Comments", "url": "https://www.reuters.com/business/a16z-backed-data-firms-fivetran-dbt-labs-merge-all-stock-deal-2025-10-13/", "title": "A16Z-backed data firms Fivetran, dbt Labs to merge in all-stock deal", "date_modified": "2025-10-13T14:42:19.000Z" }, { "content_html": "Comments", "url": "https://www.stavros.io/posts/switch-to-jujutsu-already-a-tutorial/", "title": "Switch to Jujutsu Already: A Tutorial", "date_modified": "2025-10-13T09:22:33.000Z" }, { "content_html": "Comments", "url": "https://jmmv.dev/2025/09/bazel-remote-execution.html", "title": "Trusting builds with Bazel remote execution", "date_modified": "2025-10-12T21:03:03.000Z" }, { "content_html": "

Comments

", "url": "https://www.stavros.io/posts/switch-to-jujutsu-already-a-tutorial/", "title": "Switch to Jujutsu already: a tutorial", "date_modified": "2025-10-12T08:33:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/acsandmann/rift", "title": "Show HN: Rift – A tiling window manager for macOS", "date_modified": "2025-10-12T00:22:15.000Z" }, { "content_html": "Comments", "url": "https://blog.tangled.org/intro", "title": "Tangled, a Git collaboration platform built on atproto", "date_modified": "2025-10-10T21:18:55.000Z" }, { "content_html": "Comments", "url": "https://www.githubstatus.com/incidents/k7bhmjkblcwp", "title": "GitHub Issues", "date_modified": "2025-10-09T15:05:52.000Z" }, { "content_html": "

Comments

", "url": "https://www.youtube.com/watch?v=ulJ_Pw8qqsE", "title": "Solving Git's Pain Points with Jujutsu", "date_modified": "2025-10-09T14:21:16.000Z" }, { "content_html": "Comments", "url": "https://www.geocod.io/code-and-coordinates/2025-10-02-from-millions-to-billions/", "title": "Scaling request logging with ClickHouse, Kafka, and Vector", "date_modified": "2025-10-08T09:56:25.000Z" }, { "content_html": "Comments", "url": "https://github.com/mbj/mutant", "title": "Automated code reviews via mutation testing", "date_modified": "2025-10-06T21:40:35.000Z" }, { "content_html": "

Comments

", "url": "https://landaire.net/jj-evolog/", "title": "Saving My Commit With `jj evolog`", "date_modified": "2025-10-06T16:21:09.000Z" }, { "content_html": "

At JetBrains, our mission is simple: help developers achieve engineering excellence without distractions or vendor lock-in. Today, we’re excited to share that we’re collaborating with Zed on the Agent Client Protocol (ACP) – an open protocol that lets AI coding agents work inside editors, including JetBrains IDEs, if both are ACP-compatible.

\n\n\n\n

Why this collaboration matters

\n\n\n\n\n\n\n\n

What developers get in JetBrains IDEs

\n\n\n\n\n\n\n\n

The JetBrains × Zed advantage

\n\n\n\n

Zed has quickly innovated in protocol design and seen real-world adoption. JetBrains brings decades of IDE expertise, including refactoring, debugging, navigation, and performance at scale. Together, we’re aligning on ACP-driven experiences that feel native in JetBrains IDEs while remaining open and portable across the ecosystem. The result is agents that are portable, powerful, and predictable inside your daily tools.

\n\n\n\n

What’s next

\n\n\n\n

We’re working closely with Zed on the next milestones for ACP-powered experiences in JetBrains IDEs. As work progresses, we’ll share details on early implementations, developer previews, and how partners can extend ACP-based integrations.

\n\n\n\n

Our goal hasn’t changed: empower developers with freedom of choice, clarity of control, and first-class productivity with AI agents that feel at home in JetBrains IDEs.

", "url": "https://blog.jetbrains.com/ai/2025/10/jetbrains-zed-open-interoperability-for-ai-coding-agents-in-your-ide/", "title": "JetBrains × Zed: Open Interoperability for AI Coding Agents in Your IDE", "date_modified": "2025-10-06T15:57:18.000Z" }, { "content_html": "Comments", "url": "https://github.com/jdx/mise/discussions/6564", "title": "Mise: Monorepo Tasks", "date_modified": "2025-10-06T14:07:46.000Z" }, { "content_html": "

Comments

", "url": "https://nvartolomei.com/oswald/", "title": "OSWALD - Object Storage Write-Ahead Log Device", "date_modified": "2025-10-04T12:38:07.000Z" }, { "content_html": "Comments", "url": "https://blogsystem5.substack.com/p/you-are-holding-build-files-wrong", "title": "Build files are the best tool to represent software architecture", "date_modified": "2025-10-03T21:04:00.000Z" }, { "content_html": "Comments", "url": "https://gist.github.com/tkafka/e3eb63a5ec448e9be6701bfd1f1b1e58", "title": "Detect Electron apps on Mac that hasn't been updated to fix the system wide lag", "date_modified": "2025-10-01T12:54:17.000Z" }, { "content_html": "Comments", "url": "https://www.amplifypartners.com/blog-posts/why-tigerbeetle-is-the-most-interesting-database-in-the-world", "title": "Why TigerBeetle is the most interesting database in the world", "date_modified": "2025-10-01T11:33:19.000Z" }, { "content_html": "Comments", "url": "https://cognition.ai/blog/blockdiff", "title": "Blockdiff: We built our own file format for VM disk snapshots", "date_modified": "2025-10-01T03:13:16.000Z" }, { "content_html": "

Today, we’re announcing Amazon ECS Managed Instances, a new compute option for Amazon Elastic Container Service (Amazon ECS) that enables developers to use the full range of Amazon Elastic Compute Cloud (Amazon EC2) capabilities while offloading infrastructure management responsibilities to Amazon Web Service (AWS). This new offering combines the operational simplicity of offloading infrastructure with the flexibility and control of Amazon EC2, which means customers can focus on building applications that drive innovation, while reducing total cost of ownership (TCO) and maintaining AWS best practices.

\n

Amazon ECS Managed Instances provides a fully managed container compute environment that supports a broad range of EC2 instance types and deep integration with AWS services. By default, it automatically selects the most cost-optimized EC2 instances for your workloads, but you can specify particular instance attributes or types when needed. AWS handles all aspects of infrastructure management, including provisioning, scaling, security patching, and cost optimization, enabling you to concentrate on building and running your applications.

\n

Let’s try it out

\n

Looking at the AWS Management Console experience for creating a new Amazon ECS cluster, I can see the new option for using ECS Managed Instances. Let’s take a quick tour of all the new options.

\n

\"Creating

\n

After I’ve selected Fargate and Managed Instances, I’m presented with two options. If I select Use ECS default, Amazon ECS will choose general purpose instance types based on grouping together pending Tasks, and picking the optimum instance type based on cost and resilience metrics. This is the most straightforward and recommended way to get started. Selecting Use custom – advanced opens up additional configuration parameters, where I can fine-tune the attributes of instances Amazon ECS will use.

\n

\"Creating

\n

By default, I see CPU and Memory as attributes, but I can select from 20 additional attributes to continue to filter the list of available instance types Amazon ECS can access.

\n

\"ECS

\n

After I’ve made my attribute selections, I see a list of all the instance types that match my choices.

\n

\"Creating

\n

From here, I can create my ECS cluster as usual and Amazon ECS will provision instances for me on my behalf based on the attributes and criteria I’ve defined in the previous steps.

\n

Key features of Amazon ECS Managed Instances

\n

With Amazon ECS Managed Instances, AWS takes full responsibility for infrastructure management, handling all aspects of instance provisioning, scaling, and maintenance. This includes implementing regular security patches initiated every 14 days (due to instance connection draining, the actual lifetime of the instance may be longer), with the ability to schedule maintenance windows using Amazon EC2 event windows to minimize disruption to your applications.

\n

The service provides exceptional flexibility in instance type selection. Although it automatically selects cost-optimized instance types by default, you maintain the power to specify desired instance attributes when your workloads require specific capabilities. This includes options for GPU acceleration, CPU architecture, and network performance requirements, giving you precise control over your compute environment.

\n

To help optimize costs, Amazon ECS Managed Instances intelligently manages resource utilization by automatically placing multiple tasks on larger instances when appropriate. The service continually monitors and optimizes task placement, consolidating workloads onto fewer instances to dry up, utilize and terminate idle (empty) instances, providing both high availability and cost efficiency for your containerized applications.

\n

Integration with existing AWS services is seamless, particularly with Amazon EC2 features such as EC2 pricing options. This deep integration means that you can maximize existing capacity investments while maintaining the operational simplicity of a fully managed service.

\n

Security remains a top priority with Amazon ECS Managed Instances. The service runs on Bottlerocket, a purpose-built container operating system, and maintains your security posture through automated security patches and updates. You can see all the updates and patches applied to the Bottlerocket OS image on the Bottlerocket website. This comprehensive approach to security keeps your containerized applications running in a secure, maintained environment.

\n

Available now

\n

Amazon ECS Managed Instances is available today in US East (North Virginia), US West (Oregon), Europe (Ireland), Africa (Cape Town), Asia Pacific (Singapore), and Asia Pacific (Tokyo) AWS Regions. You can start using Managed Instances through the AWS Management Console, AWS Command Line Interface (AWS CLI), or infrastructure as code (IaC) tools such as AWS Cloud Development Kit (AWS CDK) and AWS CloudFormation. You pay for the EC2 instances you use plus a management fee for the service.

\n

To learn more about Amazon ECS Managed Instances, visit the documentation and get started simplifying your container infrastructure today.

", "url": "https://aws.amazon.com/blogs/aws/announcing-amazon-ecs-managed-instances-for-containerized-applications/", "title": "Announcing Amazon ECS Managed Instances for containerized applications", "date_modified": "2025-09-30T18:46:47.000Z" }, { "content_html": "Comments", "url": "https://imbue.com/sculptor/", "title": "Show HN: Sculptor, the Missing UI for Claude Code", "date_modified": "2025-09-30T16:35:16.000Z" }, { "content_html": "Comments", "url": "https://blog.kagi.com/kagi-news", "title": "Kagi News", "date_modified": "2025-09-30T15:09:00.000Z" }, { "content_html": "Comments", "url": "https://www.greptile.com/blog/sandboxing-agents-at-the-kernel-level", "title": "Sandboxing AI Agents at the Kernel Level", "date_modified": "2025-09-29T16:40:05.000Z" }, { "content_html": "Comments", "url": "https://www.skeptrune.com/posts/use-the-accept-header-to-serve-markdown-instead-of-html-to-llms/", "title": "Use the Accept Header to Serve Markdown Instead of HTML to LLMs", "date_modified": "2025-09-28T23:33:34.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/code-mode/", "title": "Code Mode: the better way to use MCP", "date_modified": "2025-09-27T20:49:36.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=iPoL03tFBtU", "title": "Docker Was Too Slow, So We Replaced It: Nix in Production [video]", "date_modified": "2025-09-27T18:56:18.000Z" }, { "content_html": "Comments", "url": "https://idiallo.com/blog/users-only-care-about-20-percent", "title": "Users only care about 20% of your application", "date_modified": "2025-09-27T13:15:37.000Z" }, { "content_html": "

It turns out we've all been using MCP wrong.

Most agents today use MCP by directly exposing the \"tools\" to the LLM.

We tried something different: Convert the MCP tools into a TypeScript API, and then ask an LLM to write code that calls that API.

The results are striking:

  1. We found agents are able to handle many more tools, and more complex tools, when those tools are presented as a TypeScript API rather than directly. Perhaps this is because LLMs have an enormous amount of real-world TypeScript in their training set, but only a small set of contrived examples of tool calls.

  2. The approach really shines when an agent needs to string together multiple calls. With the traditional approach, the output of each tool call must feed into the LLM's neural network, just to be copied over to the inputs of the next call, wasting time, energy, and tokens. When the LLM can write code, it can skip all that, and only read back the final results it needs.

In short, LLMs are better at writing code to call MCP, than at calling MCP directly.

\n
\n

What's MCP?

\n \n \n \n
\n

For those that aren't familiar: Model Context Protocol is a standard protocol for giving AI agents access to external tools, so that they can directly perform work, rather than just chat with you.

Seen another way, MCP is a uniform way to:

MCP has been making waves throughout 2025 as it has suddenly greatly expanded the capabilities of AI agents.

The \"API\" exposed by an MCP server is expressed as a set of \"tools\". Each tool is essentially a remote procedure call (RPC) function – it is called with some parameters and returns a response. Most modern LLMs have the capability to use \"tools\" (sometimes called \"function calling\"), meaning they are trained to output text in a certain format when they want to invoke a tool. The program invoking the LLM sees this format and invokes the tool as specified, then feeds the results back into the LLM as input.

\n
\n

Anatomy of a tool call

\n \n \n \n
\n

Under the hood, an LLM generates a stream of \"tokens\" representing its output. A token might represent a word, a syllable, some sort of punctuation, or some other component of text.

A tool call, though, involves a token that does not have any textual equivalent. The LLM is trained (or, more often, fine-tuned) to understand a special token that it can output that means \"the following should be interpreted as a tool call,\" and another special token that means \"this is the end of the tool call.\" Between these two tokens, the LLM will typically write tokens corresponding to some sort of JSON message that describes the call.

For instance, imagine you have connected an agent to an MCP server that provides weather info, and you then ask the agent what the weather is like in Austin, TX. Under the hood, the LLM might generate output like the following. Note that here we've used words in <| and |> to represent our special tokens, but in fact, these tokens do not represent text at all; this is just for illustration.

I will use the Weather MCP server to find out the weather in Austin, TX.

\n 
I will use the Weather MCP server to find out the weather in Austin, TX.\n\n<|tool_call|>\n{\n  \"name\": \"get_current_weather\",\n  \"arguments\": {\n    \"location\": \"Austin, TX, USA\"\n  }\n}\n<|end_tool_call|>
\n

Upon seeing these special tokens in the output, the LLM's harness will interpret the sequence as a tool call. After seeing the end token, the harness pauses execution of the LLM. It parses the JSON message and returns it as a separate component of the structured API result. The agent calling the LLM API sees the tool call, invokes the relevant MCP server, and then sends the results back to the LLM API. The LLM's harness will then use another set of special tokens to feed the result back into the LLM:

\n 
<|tool_result|>\n{\n  \"location\": \"Austin, TX, USA\",\n  \"temperature\": 93,\n  \"unit\": \"fahrenheit\",\n  \"conditions\": \"sunny\"\n}\n<|end_tool_result|>
\n

The LLM reads these tokens in exactly the same way it would read input from the user – except that the user cannot produce these special tokens, so the LLM knows it is the result of the tool call. The LLM then continues generating output like normal.

Different LLMs may use different formats for tool calling, but this is the basic idea.

\n
\n

What's wrong with this?

\n \n \n \n
\n

The special tokens used in tool calls are things LLMs have never seen in the wild. They must be specially trained to use tools, based on synthetic training data. They aren't always that good at it. If you present an LLM with too many tools, or overly complex tools, it may struggle to choose the right one or to use it correctly. As a result, MCP server designers are encouraged to present greatly simplified APIs as compared to the more traditional API they might expose to developers.

Meanwhile, LLMs are getting really good at writing code. In fact, LLMs asked to write code against the full, complex APIs normally exposed to developers don't seem to have too much trouble with it. Why, then, do MCP interfaces have to \"dumb it down\"? Writing code and calling tools are almost the same thing, but it seems like LLMs can do one much better than the other?

The answer is simple: LLMs have seen a lot of code. They have not seen a lot of \"tool calls\". In fact, the tool calls they have seen are probably limited to a contrived training set constructed by the LLM's own developers, in order to try to train it. Whereas they have seen real-world code from millions of open source projects.

Making an LLM perform tasks with tool calling is like putting Shakespeare through a month-long class in Mandarin and then asking him to write a play in it. It's just not going to be his best work.

\n
\n

But MCP is still useful, because it is uniform

\n \n \n \n
\n

MCP is designed for tool-calling, but it doesn't actually have to be used that way.

The \"tools\" that an MCP server exposes are really just an RPC interface with attached documentation. We don't really have to present them as tools. We can take the tools, and turn them into a programming language API instead.

But why would we do that, when the programming language APIs already exist independently? Almost every MCP server is just a wrapper around an existing traditional API – why not expose those APIs?

Well, it turns out MCP does something else that's really useful: It provides a uniform way to connect to and learn about an API.

An AI agent can use an MCP server even if the agent's developers never heard of the particular MCP server, and the MCP server's developers never heard of the particular agent. This has rarely been true of traditional APIs in the past. Usually, the client developer always knows exactly what API they are coding for. As a result, every API is able to do things like basic connectivity, authorization, and documentation a little bit differently.

This uniformity is useful even when the AI agent is writing code. We'd like the AI agent to run in a sandbox such that it can only access the tools we give it. MCP makes it possible for the agentic framework to implement this, by handling connectivity and authorization in a standard way, independent of the AI code. We also don't want the AI to have to search the Internet for documentation; MCP provides it directly in the protocol.

\n
\n

OK, how does it work?

\n \n \n \n
\n

We have already extended the Cloudflare Agents SDK to support this new model!

For example, say you have an app built with ai-sdk that looks like this:

\n 
const stream = streamText({\n  model: openai(\"gpt-5\"),\n  system: \"You are a helpful assistant\",\n  messages: [\n    { role: \"user\", content: \"Write a function that adds two numbers\" }\n  ],\n  tools: {\n    // tool definitions \n  }\n})
\n

You can wrap the tools and prompt with the codemode helper, and use them in your app: 

\n 
import { codemode } from \"agents/codemode/ai\";\n\nconst {system, tools} = codemode({\n  system: \"You are a helpful assistant\",\n  tools: {\n    // tool definitions \n  },\n  // ...config\n})\n\nconst stream = streamText({\n  model: openai(\"gpt-5\"),\n  system,\n  tools,\n  messages: [\n    { role: \"user\", content: \"Write a function that adds two numbers\" }\n  ]\n})
\n

With this change, your app will now start generating and running code that itself will make calls to the tools you defined, MCP servers included. We will introduce variants for other libraries in the very near future. Read the docs for more details and examples. 

\n
\n

Converting MCP to TypeScript

\n \n \n \n
\n

When you connect to an MCP server in \"code mode\", the Agents SDK will fetch the MCP server's schema, and then convert it into a TypeScript API, complete with doc comments based on the schema.

For example, connecting to the MCP server at gitmcp.io/cloudflare/agents, will generate a TypeScript definition like this:

\n 
interface FetchAgentsDocumentationInput {\n  [k: string]: unknown;\n}\ninterface FetchAgentsDocumentationOutput {\n  [key: string]: any;\n}\n\ninterface SearchAgentsDocumentationInput {\n  /**\n   * The search query to find relevant documentation\n   */\n  query: string;\n}\ninterface SearchAgentsDocumentationOutput {\n  [key: string]: any;\n}\n\ninterface SearchAgentsCodeInput {\n  /**\n   * The search query to find relevant code files\n   */\n  query: string;\n  /**\n   * Page number to retrieve (starting from 1). Each page contains 30\n   * results.\n   */\n  page?: number;\n}\ninterface SearchAgentsCodeOutput {\n  [key: string]: any;\n}\n\ninterface FetchGenericUrlContentInput {\n  /**\n   * The URL of the document or page to fetch\n   */\n  url: string;\n}\ninterface FetchGenericUrlContentOutput {\n  [key: string]: any;\n}\n\ndeclare const codemode: {\n  /**\n   * Fetch entire documentation file from GitHub repository:\n   * cloudflare/agents. Useful for general questions. Always call\n   * this tool first if asked about cloudflare/agents.\n   */\n  fetch_agents_documentation: (\n    input: FetchAgentsDocumentationInput\n  ) => Promise<FetchAgentsDocumentationOutput>;\n\n  /**\n   * Semantically search within the fetched documentation from\n   * GitHub repository: cloudflare/agents. Useful for specific queries.\n   */\n  search_agents_documentation: (\n    input: SearchAgentsDocumentationInput\n  ) => Promise<SearchAgentsDocumentationOutput>;\n\n  /**\n   * Search for code within the GitHub repository: \"cloudflare/agents\"\n   * using the GitHub Search API (exact match). Returns matching files\n   * for you to query further if relevant.\n   */\n  search_agents_code: (\n    input: SearchAgentsCodeInput\n  ) => Promise<SearchAgentsCodeOutput>;\n\n  /**\n   * Generic tool to fetch content from any absolute URL, respecting\n   * robots.txt rules. Use this to retrieve referenced urls (absolute\n   * urls) that were mentioned in previously fetched documentation.\n   */\n  fetch_generic_url_content: (\n    input: FetchGenericUrlContentInput\n  ) => Promise<FetchGenericUrlContentOutput>;\n};
\n

This TypeScript is then loaded into the agent's context. Currently, the entire API is loaded, but future improvements could allow an agent to search and browse the API more dynamically – much like an agentic coding assistant would.

\n
\n

Running code in a sandbox

\n \n \n \n
\n

Instead of being presented with all the tools of all the connected MCP servers, our agent is presented with just one tool, which simply executes some TypeScript code.

The code is then executed in a secure sandbox. The sandbox is totally isolated from the Internet. Its only access to the outside world is through the TypeScript APIs representing its connected MCP servers.

These APIs are backed by RPC invocation which calls back to the agent loop. There, the Agents SDK dispatches the call to the appropriate MCP server.

The sandboxed code returns results to the agent in the obvious way: by invoking console.log(). When the script finishes, all the output logs are passed back to the agent.

\n
\n \n
\n
\n

Dynamic Worker loading: no containers here

\n \n \n \n
\n

This new approach requires access to a secure sandbox where arbitrary code can run. So where do we find one? Do we have to run containers? Is that expensive?

No. There are no containers. We have something much better: isolates.

The Cloudflare Workers platform has always been based on V8 isolates, that is, isolated JavaScript runtimes powered by the V8 JavaScript engine.

Isolates are far more lightweight than containers. An isolate can start in a handful of milliseconds using only a few megabytes of memory.

Isolates are so fast that we can just create a new one for every piece of code the agent runs. There's no need to reuse them. There's no need to prewarm them. Just create it, on demand, run the code, and throw it away. It all happens so fast that the overhead is negligible; it's almost as if you were just eval()ing the code directly. But with security.

\n
\n

The Worker Loader API

\n \n \n \n
\n

Until now, though, there was no way for a Worker to directly load an isolate containing arbitrary code. All Worker code instead had to be uploaded via the Cloudflare API, which would then deploy it globally, so that it could run anywhere. That's not what we want for Agents! We want the code to just run right where the agent is.

To that end, we've added a new API to the Workers platform: the Worker Loader API. With it, you can load Worker code on-demand. Here's what it looks like:

\n 
// Gets the Worker with the given ID, creating it if no such Worker exists yet.\nlet worker = env.LOADER.get(id, async () => {\n  // If the Worker does not already exist, this callback is invoked to fetch\n  // its code.\n\n  return {\n    compatibilityDate: \"2025-06-01\",\n\n    // Specify the worker's code (module files).\n    mainModule: \"foo.js\",\n    modules: {\n      \"foo.js\":\n        \"export default {\\n\" +\n        \"  fetch(req, env, ctx) { return new Response('Hello'); }\\n\" +\n        \"}\\n\",\n    },\n\n    // Specify the dynamic Worker's environment (`env`).\n    env: {\n      // It can contain basic serializable data types...\n      SOME_NUMBER: 123,\n\n      // ... and bindings back to the parent worker's exported RPC\n      // interfaces, using the new `ctx.exports` loopback bindings API.\n      SOME_RPC_BINDING: ctx.exports.MyBindingImpl({props})\n    },\n\n    // Redirect the Worker's `fetch()` and `connect()` to proxy through\n    // the parent worker, to monitor or filter all Internet access. You\n    // can also block Internet access completely by passing `null`.\n    globalOutbound: ctx.exports.OutboundProxy({props}),\n  };\n});\n\n// Now you can get the Worker's entrypoint and send requests to it.\nlet defaultEntrypoint = worker.getEntrypoint();\nawait defaultEntrypoint.fetch(\"http://example.com\");\n\n// You can get non-default entrypoints as well, and specify the\n// `ctx.props` value to be delivered to the entrypoint.\nlet someEntrypoint = worker.getEntrypoint(\"SomeEntrypointClass\", {\n  props: {someProp: 123}\n});
\n

You can start playing with this API right now when running workerd locally with Wrangler (check out the docs), and you can sign up for beta access to use it in production.

\n
\n

Workers are better sandboxes

\n \n \n \n
\n

The design of Workers makes it unusually good at sandboxing, especially for this use case, for a few reasons:

\n
\n

Faster, cheaper, disposable sandboxes

\n \n \n \n
\n

The Workers platform uses isolates instead of containers. Isolates are much lighter-weight and faster to start up. It takes mere milliseconds to start a fresh isolate, and it's so cheap we can just create a new one for every single code snippet the agent generates. There's no need to worry about pooling isolates for reuse, prewarming, etc.

We have not yet finalized pricing for the Worker Loader API, but because it is based on isolates, we will be able to offer it at a significantly lower cost than container-based solutions.

\n
\n

Isolated by default, but connected with bindings

\n \n \n \n
\n

Workers are just better at handling isolation.

In Code Mode, we prohibit the sandboxed worker from talking to the Internet. The global fetch() and connect() functions throw errors.

But on most platforms, this would be a problem. On most platforms, the way you get access to private resources is, you start with general network access. Then, using that network access, you send requests to specific services, passing them some sort of API key to authorize private access.

But Workers has always had a better answer. In Workers, the \"environment\" (env object) doesn't just contain strings, it contains live objects, also known as \"bindings\". These objects can provide direct access to private resources without involving generic network requests.

In Code Mode, we give the sandbox access to bindings representing the MCP servers it is connected to. Thus, the agent can specifically access those MCP servers without having network access in general.

Limiting access via bindings is much cleaner than doing it via, say, network-level filtering or HTTP proxies. Filtering is hard on both the LLM and the supervisor, because the boundaries are often unclear: the supervisor may have a hard time identifying exactly what traffic is legitimately necessary to talk to an API. Meanwhile, the LLM may have difficulty guessing what kinds of requests will be blocked. With the bindings approach, it's well-defined: the binding provides a JavaScript interface, and that interface is allowed to be used. It's just better this way.

\n
\n

No API keys to leak

\n \n \n \n
\n

An additional benefit of bindings is that they hide API keys. The binding itself provides an already-authorized client interface to the MCP server. All calls made on it go to the agent supervisor first, which holds the access tokens and adds them into requests sent on to MCP.

This means that the AI cannot possibly write code that leaks any keys, solving a common security problem seen in AI-authored code today.

\n
\n

Try it now!

\n \n \n \n
\n \n
\n

Sign up for the production beta

\n \n \n \n
\n

The Dynamic Worker Loader API is in closed beta. To use it in production, sign up today.

\n
\n

Or try it locally

\n \n \n \n
\n

If you just want to play around, though, Dynamic Worker Loading is fully available today when developing locally with Wrangler and workerd – check out the docs for Dynamic Worker Loading and code mode in the Agents SDK to get started.

", "url": "https://blog.cloudflare.com/code-mode/", "title": "Code Mode: the better way to use MCP", "date_modified": "2025-09-26T13:00:00.000Z" }, { "content_html": "Comments", "url": "https://news.sky.com/video/all-british-adults-to-require-a-digital-id-brit-card-13438041", "title": "All British adults to require a digital ID 'Brit Card'", "date_modified": "2025-09-26T02:09:47.000Z" }, { "content_html": "Comments", "url": "https://ollama.com/blog/web-search", "title": "Ollama Web Search", "date_modified": "2025-09-25T19:21:52.000Z" }, { "content_html": "Comments", "url": "https://meks.quest/blogs/the-theatre-of-pull-requests-and-code-review", "title": "The Theatre of Pull Requests and Code Review", "date_modified": "2025-09-25T10:35:19.000Z" }, { "content_html": "Comments", "url": "https://zed.dev/blog/pricing-change-llm-usage-is-now-token-based", "title": "Zed's Pricing Has Changed: LLM Usage Is Now Token-Based", "date_modified": "2025-09-24T16:13:49.000Z" }, { "content_html": "Comments", "url": "https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/ace-fca.md", "title": "Getting AI to work in complex codebases", "date_modified": "2025-09-23T14:27:36.000Z" }, { "content_html": "Comments", "url": "https://mitchellh.com/writing/libghostty-is-coming", "title": "Libghostty is coming", "date_modified": "2025-09-23T13:56:28.000Z" }, { "content_html": "Comments", "url": "https://marcobambini.substack.com/p/why-local-first-apps-havent-become", "title": "Why haven't local-first apps become popular?", "date_modified": "2025-09-22T13:17:59.000Z" }, { "content_html": "Comments", "url": "https://blog.val.town/vtlsp", "title": "Building a better online editor for TypeScript", "date_modified": "2025-09-22T04:22:32.000Z" }, { "content_html": "

Comments

", "url": "https://michael.stapelberg.ch/posts/2025-09-21-nixcon-2025-trip-report/", "title": "NixCon 2025 Trip Report", "date_modified": "2025-09-21T07:51:09.000Z" }, { "content_html": "Comments", "url": "https://www.seangoedecke.com/ai-agents-and-code-review/", "title": "If you are good at code review, you will be good at using AI agents", "date_modified": "2025-09-20T04:59:10.000Z" }, { "content_html": "Comments", "url": "https://www.xtxmarkets.com/tech/2025-ternfs/", "title": "TernFS – An exabyte scale, multi-region distributed filesystem", "date_modified": "2025-09-18T14:36:44.000Z" }, { "content_html": "Comments", "url": "https://stategraph.dev/blog/why-stategraph/", "title": "Stategraph: Terraform state as a distributed systems problem", "date_modified": "2025-09-17T08:38:17.000Z" }, { "content_html": "Comments", "url": "https://blog.asciinema.org/post/three-point-o/", "title": "Asciinema CLI 3.0 rewritten in Rust, adds live streaming, upgrades file format", "date_modified": "2025-09-15T16:06:30.000Z" }, { "content_html": "

Comments

", "url": "https://youtu.be/hdBm4K-vvt0", "title": "Waterpark: Transforming Healthcare with Distributed Actors", "date_modified": "2025-09-15T13:55:45.000Z" }, { "content_html": "Comments", "url": "https://haystackeditor.com", "title": "Show HN: Haystack – Review pull requests like you wrote them yourself", "date_modified": "2025-09-10T18:21:12.000Z" }, { "content_html": "Comments", "url": "https://mergify.com/blog/the-origin-story-of-merge-queues", "title": "The Origin Story of Merge Queues", "date_modified": "2025-09-10T15:43:43.000Z" }, { "content_html": "

Comments

", "url": "https://mergify.com/blog/the-origin-story-of-merge-queues", "title": "The Origin Story of Merge Queues", "date_modified": "2025-09-10T15:43:25.000Z" }, { "content_html": "Comments", "url": "https://til.andrew-quinn.me/posts/tarsnap-is-cozy/", "title": "Tarsnap is cozy", "date_modified": "2025-09-10T12:17:45.000Z" }, { "content_html": "Comments", "url": "https://github.com/Barre/ZeroFS", "title": "Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage", "date_modified": "2025-09-08T22:13:04.000Z" }, { "content_html": "Comments", "url": "https://github.com/BeaconBay/ck", "title": "Show HN: Semantic grep with local embeddings", "date_modified": "2025-09-07T11:20:37.000Z" }, { "content_html": "Comments", "url": "https://bigtechpr.substack.com/p/why-browser-co-610m-is-cheap", "title": "Why Browser Company at $610M is cheap", "date_modified": "2025-09-05T14:38:07.000Z" }, { "content_html": "Comments", "url": "https://codesmash.dev/why-i-ditched-docker-for-podman-and-you-should-too", "title": "I Ditched Docker for Podman (and You Should Too)", "date_modified": "2025-09-05T11:56:59.000Z" }, { "content_html": "

Comments

", "url": "https://radicle.xyz/2025/09/04/radicle-1.4.0", "title": "Radicle 1.4.0 – Lily", "date_modified": "2025-09-05T07:03:08.000Z" }, { "content_html": "Comments", "url": "https://github.com/browseros-ai/BrowserOS/issues/99", "title": "Ask HN: What Arc/Dia features should we prioritize?", "date_modified": "2025-09-04T16:13:53.000Z" }, { "content_html": "Comments", "url": "https://www.atlassian.com/blog/announcements/atlassian-acquires-the-browser-company", "title": "The Browser Company (Arc, Dia) Has Been Acquired by Atlassian", "date_modified": "2025-09-04T12:12:31.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/jj-vcs/jj/releases/tag/v0.33.0", "title": "jujutsu v0.33.0 released", "date_modified": "2025-09-04T09:21:38.000Z" }, { "content_html": "Comments", "url": "https://www.warp.dev/blog/introducing-warp-code-prompt-to-prod", "title": "Warp Code: the fastest way from prompt to production", "date_modified": "2025-09-03T15:31:31.000Z" }, { "content_html": "Comments", "url": "https://zed.dev/blog/claude-code-via-acp", "title": "Claude Code: Now in Beta in Zed", "date_modified": "2025-09-03T15:07:20.000Z" }, { "content_html": "Comments", "url": "https://morningcoffee.io/parallel-ai-agents-are-a-game-changer.html", "title": "Parallel AI agents are a game changer", "date_modified": "2025-09-02T22:44:50.000Z" }, { "content_html": "Comments", "url": "https://www.sanity.io/blog/first-attempt-will-be-95-garbage", "title": "A staff engineer's journey with Claude Code", "date_modified": "2025-09-02T19:34:24.000Z" }, { "content_html": "Comments", "url": "https://jj-for-everyone.github.io/", "title": "Jujutsu for Everyone", "date_modified": "2025-08-31T15:31:04.000Z" }, { "content_html": "Comments", "url": "https://jj-for-everyone.github.io/", "title": "Jujutsu for everyone", "date_modified": "2025-08-31T15:31:04.000Z" }, { "content_html": "

Comments

", "url": "https://jj-for-everyone.github.io/", "title": "Jujutsu for everyone", "date_modified": "2025-08-31T15:29:08.000Z" }, { "content_html": "

Comments

", "url": "https://aws.amazon.com/blogs/compute/building-resilient-multi-tenant-systems-with-amazon-sqs-fair-queues/", "title": "AWS has finally made SQS a viable queuing solution", "date_modified": "2025-08-30T15:56:09.000Z" }, { "content_html": "Comments", "url": "https://agentclientprotocol.com/overview/introduction", "title": "Agent Client Protocol", "date_modified": "2025-08-30T12:42:30.000Z" }, { "content_html": "Comments", "url": "https://marcoapp.io/blog/offline-first-landscape", "title": "Offline-First Landscape – 2025", "date_modified": "2025-08-29T16:20:31.000Z" }, { "content_html": "Comments", "url": "https://github.com/DeluxeOwl/chronicle", "title": "Chronicle – Idiomatic, type safe event sourcing framework for Go", "date_modified": "2025-08-28T16:37:02.000Z" }, { "content_html": "Comments", "url": "https://claude-checkpoints.com/", "title": "Claude Code Checkpoints", "date_modified": "2025-08-28T09:16:17.000Z" }, { "content_html": "Comments", "url": "https://monodraw.helftone.com/", "title": "Monodraw", "date_modified": "2025-08-27T10:54:41.000Z" }, { "content_html": "Comments", "url": "https://github.com/orgs/community/discussions/170758", "title": "The GitHub website is slow on Safari", "date_modified": "2025-08-27T09:43:43.000Z" }, { "content_html": "Comments", "url": "https://poor.dev/blog/building-zellij-web-terminal/", "title": "Terminal sessions you can bookmark", "date_modified": "2025-08-27T07:16:46.000Z" }, { "content_html": "Comments", "url": "https://jakub.kr/components/oklch-colors", "title": "What are OKLCH colors?", "date_modified": "2025-08-25T06:32:04.000Z" }, { "content_html": "Comments", "url": "https://github.com/mbuhot/glyn", "title": "Glyn: Type-safe PubSub and Registry for Gleam actors with distributed clustering", "date_modified": "2025-08-22T22:29:28.000Z" }, { "content_html": "Comments", "url": "https://git.vuxu.org/nitro/about/", "title": "Nitro: A tiny but flexible init system and process supervisor", "date_modified": "2025-08-22T19:06:29.000Z" }, { "content_html": "Comments", "url": "https://devenv.sh/blog/2025/08/22/closing-the-nix-gap-from-environments-to-packaged-applications-for-rust/", "title": "Closing the Nix gap: From environments to packaged applications for rust", "date_modified": "2025-08-22T16:06:15.000Z" }, { "content_html": "

Comments

", "url": "https://optique.dev/", "title": "Optique: Type-safe combinatorial CLI parser for TypeScript", "date_modified": "2025-08-21T03:08:51.000Z" }, { "content_html": "Comments", "url": "https://tigerbeetle.com/blog/2025-08-04-code-review-can-be-better/", "title": "Code review can be better", "date_modified": "2025-08-20T23:10:37.000Z" }, { "content_html": "

Comments

", "url": "https://www.pauladamsmith.com/blog/2025/08/jj-absorb.html", "title": "Let `jj absorb` help you keep a clean commit history", "date_modified": "2025-08-20T21:04:23.000Z" }, { "content_html": "Comments", "url": "https://github.com/zedless-editor/zed", "title": "Zedless: Zed fork focused on privacy and being local-first", "date_modified": "2025-08-20T18:47:03.000Z" }, { "content_html": "Comments", "url": "https://www.lastweekinaws.com/blog/aws-in-2025-the-stuff-you-think-you-know-thats-now-wrong/", "title": "AWS in 2025: Stuff you think you know that's now wrong", "date_modified": "2025-08-20T15:30:07.000Z" }, { "content_html": "Comments", "url": "https://zed.dev/blog/sequoia-backs-zed", "title": "Sequoia backs Zed", "date_modified": "2025-08-20T12:13:16.000Z" }, { "content_html": "

Comments

", "url": "https://tigerbeetle.com/blog/2025-08-04-code-review-can-be-better/", "title": "Code Review Can Be Better", "date_modified": "2025-08-20T11:03:02.000Z" }, { "content_html": "Comments", "url": "https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-misdirected/", "title": "Modern CI is too complex and misdirected (2021)", "date_modified": "2025-08-20T03:30:06.000Z" }, { "content_html": "

Comments

", "url": "https://lore.kernel.org/git/20250819-b4-pks-history-builtin-v1-0-9b77c32688fe@pks.im/", "title": "Git RFC: Introduce jj-inspired git-history(1) command for easy history editing", "date_modified": "2025-08-20T00:38:56.000Z" }, { "content_html": "Comments", "url": "https://agents.md/", "title": "AGENTS.md – Open format for guiding coding agents", "date_modified": "2025-08-20T00:15:03.000Z" }, { "content_html": "Comments", "url": "https://research.kudelskisecurity.com/2025/08/19/how-we-exploited-coderabbit-from-a-simple-pr-to-rce-and-write-access-on-1m-repositories/", "title": "How we exploited CodeRabbit: From simple PR to RCE and write access on 1M repos", "date_modified": "2025-08-19T15:55:15.000Z" }, { "content_html": "Comments", "url": "https://www.theverge.com/news/761240/uk-apple-us-encryption-back-door-demands-dropped", "title": "UK drops demand for backdoor into Apple encryption", "date_modified": "2025-08-19T11:58:34.000Z" }, { "content_html": "

Comments

", "url": "https://fzakaria.com/2025/08/17/using-nix-as-a-library", "title": "Using Nix as a library", "date_modified": "2025-08-18T21:01:40.000Z" }, { "content_html": "Comments", "url": "https://trychroma.com/cloud", "title": "Show HN: Chroma Cloud – serverless search database for AI", "date_modified": "2025-08-18T19:20:01.000Z" }, { "content_html": "Comments", "url": "https://roguesecurity.dev/blog/systemd-hardening", "title": "SystemD Service Hardening", "date_modified": "2025-08-18T04:57:47.000Z" }, { "content_html": "Comments", "url": "https://goose.icu/lambda/", "title": "Eliminating JavaScript cold starts on AWS Lambda", "date_modified": "2025-08-16T11:44:42.000Z" }, { "content_html": "Comments", "url": "https://tylercipriani.com/blog/2025/08/15/git-lfs/", "title": "The Future of Large Files in Git Is Git", "date_modified": "2025-08-15T20:07:06.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/actions/runner/blob/v2.328.0/src/Misc/layoutroot/safe_sleep.sh", "title": "GitHub Actions safe_sleep.sh", "date_modified": "2025-08-15T10:07:53.000Z" }, { "content_html": "

Comments

", "url": "https://radicle.xyz/2025/08/14/jujutsu-with-radicle", "title": "Jujutsu with Radicle", "date_modified": "2025-08-14T18:22:58.000Z" }, { "content_html": "Comments", "url": "https://astral.sh/blog/introducing-pyx", "title": "PYX: The next step in Python packaging", "date_modified": "2025-08-13T18:42:49.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=44876289", "title": "Ask HN: What alternatives to GitHub are you using?", "date_modified": "2025-08-12T13:59:34.000Z" }, { "content_html": "Comments", "url": "https://radicle.xyz/2025/08/12/radicle-1.3.0", "title": "Radicle 1.3.0", "date_modified": "2025-08-12T11:33:42.000Z" }, { "content_html": "

Comments

", "url": "https://commitmono.com/", "title": "Commit Mono", "date_modified": "2025-08-11T17:28:47.000Z" }, { "content_html": "Comments", "url": "https://www.theverge.com/news/757461/microsoft-github-thomas-dohmke-resignation-coreai-team-transition", "title": "GitHub is no longer independent at Microsoft after CEO resignation", "date_modified": "2025-08-11T15:47:48.000Z" }, { "content_html": "Comments", "url": "https://github.blog/news-insights/company-news/goodbye-github/", "title": "Auf Wiedersehen, GitHub – CEO Steps Down", "date_modified": "2025-08-11T15:01:39.000Z" }, { "content_html": "

Comments

", "url": "https://www.youtube.com/watch?v=63Z2oNW1Bf4", "title": "Gleam’s Interoperability with Erlang and Elixir", "date_modified": "2025-08-11T10:31:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/liquidmetal-dev/flintlock", "title": "Flintlock – Create and manage the lifecycle of MicroVMs, backed by containerd", "date_modified": "2025-08-10T15:30:48.000Z" }, { "content_html": "

Comments

", "url": "https://yoyo-code.com/why-is-github-ui-getting-so-much-slower/", "title": "Why is GitHub UI getting so much slower?", "date_modified": "2025-08-09T02:44:20.000Z" }, { "content_html": "Comments", "url": "https://bytemash.net/posts/i-went-down-the-linear-rabbit-hole/", "title": "Linear sent me down a local-first rabbit hole", "date_modified": "2025-08-08T05:45:42.000Z" }, { "content_html": "Comments", "url": "https://cursor.com/cli", "title": "Cursor CLI", "date_modified": "2025-08-07T20:53:54.000Z" }, { "content_html": "Comments", "url": "https://tomscii.sig7.se/2024/01/Ditching-GitHub", "title": "Ditching GitHub (2024)", "date_modified": "2025-08-07T16:18:40.000Z" }, { "content_html": "Comments", "url": "https://github.com/gleam-lang/gleam/blob/main/changelog/v1.12.md", "title": "Gleam v1.12.0 Released", "date_modified": "2025-08-06T17:57:57.000Z" }, { "content_html": "Comments", "url": "https://github.com/gleam-lang/gleam/blob/main/changelog/v1.12.md", "title": "Gleam v1.12", "date_modified": "2025-08-06T17:57:57.000Z" }, { "content_html": "Comments", "url": "https://yoyo-code.com/why-is-github-ui-getting-so-much-slower/", "title": "Why is GitHub UI getting so much slower?", "date_modified": "2025-08-05T16:07:46.000Z" }, { "content_html": "Comments", "url": "https://yoyo-code.com/why-is-github-ui-getting-so-much-slower/", "title": "Why is GitHub UI getting slower?", "date_modified": "2025-08-05T16:07:46.000Z" }, { "content_html": "Comments", "url": "https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/", "title": "Genie 3: A new frontier for world models", "date_modified": "2025-08-05T14:08:52.000Z" }, { "content_html": "Comments", "url": "https://apps.apple.com/cn/app/ublock-origin-lite/id6745342698", "title": "uBlock Origin Lite now available for Safari", "date_modified": "2025-08-05T09:01:54.000Z" }, { "content_html": "Comments", "url": "https://underjord.io/500-virtual-linux-devices-on-arm64.html", "title": "Virtual Linux Devices on ARM64", "date_modified": "2025-08-04T11:51:00.000Z" }, { "content_html": "Comments", "url": "https://automerge.org/blog/automerge-3/", "title": "Automerge 3.0", "date_modified": "2025-08-03T15:08:58.000Z" }, { "content_html": "

Comments

", "url": "https://maddie.wtf/posts/2025-07-21-jujutsu-for-busy-devs/entry/1", "title": "Jujutsu For Busy Devs, Part 2: \"How Do I...?\"", "date_modified": "2025-08-02T04:26:53.000Z" }, { "content_html": "Comments", "url": "https://streamnative.io/products/ursa", "title": "Ursa: A leaderless, object storage–based alternative to Kafka", "date_modified": "2025-07-31T15:01:08.000Z" }, { "content_html": "Comments", "url": "https://codspeed.io/blog/benchmarks-in-ci-without-noise", "title": "Benchmarks in CI: Escaping the Cloud Chaos", "date_modified": "2025-07-31T12:40:05.000Z" }, { "content_html": "

Comments

", "url": "https://sagikazarmark.hu/blog/vanity-import-paths-in-go/", "title": "Vanity import paths in Go (2019)", "date_modified": "2025-07-31T12:32:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/jkoppel/jj-workshop", "title": "The JJ VCS workshop: A zero-to-hero speedrun", "date_modified": "2025-07-27T12:23:31.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/jkoppel/jj-workshop", "title": "The JJ VCS Workshop: A Zero-to-Hero Speedrun", "date_modified": "2025-07-27T12:22:19.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/how-anthropic-teams-use-claude-code", "title": "How Anthropic teams use Claude Code", "date_modified": "2025-07-25T01:43:27.000Z" }, { "content_html": "Comments", "url": "https://buf.build/blog/hyperpb", "title": "Hyperpb: 10x faster dynamic Protobuf parsing that's faster than generated code", "date_modified": "2025-07-23T17:32:37.000Z" }, { "content_html": "Comments", "url": "https://blog.ydb.tech/the-surprising-grpc-client-bottleneck-in-low-latency-networks-and-how-to-get-around-it-69d6977a1d02", "title": "The Surprising gRPC Client Bottleneck in Low-Latency Networks", "date_modified": "2025-07-23T13:23:20.000Z" }, { "content_html": "Comments", "url": "https://radicle.xyz/2025/07/23/using-radicle-ci-for-development", "title": "Using Radicle CI", "date_modified": "2025-07-23T13:09:01.000Z" }, { "content_html": "Comments", "url": "https://plane.so/blog/everything-you-need-to-know-about-plane-air-gapped", "title": "We built an air-gapped Jira alternative for regulated industries", "date_modified": "2025-07-22T19:17:32.000Z" }, { "content_html": "Comments", "url": "https://maddie.wtf/posts/2025-07-21-jujutsu-for-busy-devs", "title": "Jujutsu for busy devs", "date_modified": "2025-07-22T00:21:48.000Z" }, { "content_html": "Comments", "url": "https://github.com/alefore/duende", "title": "Show HN: Duende: Web UX for guiding Gemini as it improves your source code", "date_modified": "2025-07-20T20:41:57.000Z" }, { "content_html": "Comments", "url": "https://clickhouse.com/blog/postgres-to-clickhouse-data-modeling-tips-v2", "title": "Postgres to ClickHouse: Data Modeling Tips", "date_modified": "2025-07-19T17:24:32.000Z" }, { "content_html": "Comments", "url": "https://www.notcheckmark.com/2025/07/rethinking-cli-interfaces-for-ai/", "title": "Rethinking CLI interfaces for AI", "date_modified": "2025-07-19T16:58:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/ryoppippi/ccusage", "title": "Ccusage: A CLI tool for analyzing Claude Code usage from local JSONL files", "date_modified": "2025-07-18T23:22:49.000Z" }, { "content_html": "Comments", "url": "https://conductor.build/", "title": "Show HN: Conductor, a Mac app that lets you run a bunch of Claude Codes at once", "date_modified": "2025-07-17T15:43:04.000Z" }, { "content_html": "Comments", "url": "https://neon.com/blog/an-apology-and-a-recap-on-may-june-stability", "title": "A Recap on May/June Stability at Neon", "date_modified": "2025-07-16T20:22:57.000Z" }, { "content_html": "Comments", "url": "https://drew.silcock.dev/blog/artisanal-git/", "title": "Artisanal Handcrafted Git Repositories", "date_modified": "2025-07-16T19:45:24.000Z" }, { "content_html": "Comments", "url": "https://drew.silcock.dev/blog/artisanal-git/", "title": "Artisanal handcrafted Git repositories", "date_modified": "2025-07-16T19:45:24.000Z" }, { "content_html": "Comments", "url": "https://www.instantdb.com/essays/agents", "title": "How and where will agents ship software?", "date_modified": "2025-07-16T17:47:08.000Z" }, { "content_html": "

Amazon S3 Metadata now provides complete visibility into all your existing objects in your Amazon Simple Storage Service (Amazon S3) buckets, expanding beyond new objects and changes. With this expanded coverage, you can analyze and query metadata for your entire S3 storage footprint.

\n

Today, many customers rely on Amazon S3 to store unstructured data at scale. To understand what’s in a bucket, you often need to build and maintain custom systems that scan for objects, track changes, and manage metadata over time. These systems are expensive to maintain and hard to keep up to date as data grows.

\n

Since the launch of S3 Metadata at re:Invent 2024, you’ve been able to query new and updated object metadata using metadata tables instead of relying on Amazon S3 Inventory or object-level APIs such as ListObjects, HeadObject, and GetObject—which can introduce latency and impact downstream workflows.

\n

To make it easier for you to work with this expanded metadata, S3 Metadata introduces live inventory tables that work with familiar SQL-based tools. After your existing objects are backfilled into the system, any updates like uploads or deletions typically appear within an hour in your live inventory tables.

\n

With S3 Metadata live inventory tables, you get a fully managed Apache Iceberg table that provides a complete and current snapshot of the objects and their metadata in your bucket, including existing objects, thanks to backfill support. These tables are refreshed automatically within an hour of changes such as uploads or deletions, so you stay up to date. You can use them to identify objects with specific properties—like unencrypted data, missing tags, or particular storage classes—and to support analytics, cost optimization, auditing, and governance.

\n

S3 Metadata journal tables, previously known as S3 Metadata tables, are automatically enabled when you configure live inventory tables, provide a near real-time view of object-level changes in your bucket—including uploads, deletions, and metadata updates. These tables are ideal for auditing activity, tracking the lifecycle of objects, and generating event-driven insights. For example, you can use them to find out which objects were deleted in the past 24 hours, identify the requester making the most PUT operations, or monitor updates to object metadata over time.

\n

S3 Metadata tables are created in a namespace name that is similar to your bucket name for easier discovery. The tables are stored in AWS table buckets, grouped by account and Region. After you enable S3 Metadata for a general purpose S3 bucket, the system creates and maintains these tables for you. You don’t need to manage compaction or garbage collection processes—S3 Tables takes care of table maintenance tasks in the background.

\n

These new tables help avoid waiting for metadata discovery before processing can begin, making them ideal for large-scale analytics and machine learning (ML) workloads. By querying metadata ahead of time, you can schedule GPU jobs more efficiently and reduce idle time in compute-intensive environments.

\n

Let’s see how it works
To see how this works in practice, I configure S3 Metadata for a general purpose bucket using the AWS Management Console.

\n

\"S3

\n

After choosing a general purpose bucket, I choose the Metadata tab, then I choose Create metadata configuration.

\n

\"S3For Journal table, I can choose the Server-side encryption option and the Record expiration period. For Live Inventory table, I choose Enabled and I can select the Server-side encryption options.

\n

I configure Record expiration on the journal table. Journal table records expire after the specified number of days, 365 days (one year) in my example.

\n

Then, I choose Create metadata configuration.

\n

\n

S3 Metadata creates the live inventory table and journal table. In the Live Inventory table section, I can observe the Table status: the system immediately starts to backfill the table with existing object metadata. It can take between minutes to hours. The exact time depends on the quantity of objects you have in your S3 bucket.

\n

While waiting, I also upload and delete objects to generate data in the journal table.

\n

Then, I navigate to Amazon Athena to start querying the new tables.

\n

\n

I choose Query table with Athena to start querying the table. I can choose between a couple of default queries on the console.

\n

\"MetadataBlog-rev3\"

\n

In Athena, I observe the structure of the tables in the AWSDataCatalog Data source and I start with a short query to check how many records are available in the journal table. I already have 6,488 entries:

\n
SELECT count(*) FROM \"b_aws-news-blog-metadata-inventory\".\"journal\";\n\n# _col0\n1 6488
\n

Here are a couple of example queries I tried on the journal table:

\n
# Query deleted objects in last 24 hours\n# Use is_delete_marker=true for versioned buckets and record_type='DELETE' otherwise\nSELECT bucket, key, version_id, last_modified_date\nFROM \"s3tablescatalog/aws-s3\".\"b_aws-news-blog-metadata-inventory\".\"journal\"\nWHERE last_modified_date >= (current_date - interval '1' day) AND is_delete_marker = true;\n\n# bucket key version_id last_modified_date is_delete_marker\n1 aws-news-blog-metadata-inventory .build/index-build/arm64-apple-macosx/debug/index/store/v5/records/G0/NSURLSession.h-JET61D329FG0 \n2 aws-news-blog-metadata-inventory .build/index-build/arm64-apple-macosx/debug/index/store/v5/records/G5/cdefs.h-PJ21EUWKMWG5 \n3 aws-news-blog-metadata-inventory .build/index-build/arm64-apple-macosx/debug/index/store/v5/records/FX/buf.h-25EDY57V6ZXFX \n4 aws-news-blog-metadata-inventory .build/index-build/arm64-apple-macosx/debug/index/store/v5/records/G6/NSMeasurementFormatter.h-3FN8J9CLVMYG6 \n5 aws-news-blog-metadata-inventory .build/index-build/arm64-apple-macosx/debug/index/store/v5/records/G8/NSXMLDocument.h-1UO2NUJK0OAG8 \n\n# Query recent PUT requests IP addresses\nSELECT source_ip_address, count(source_ip_address)\nFROM \"s3tablescatalog/aws-s3\".\"b_aws-news-blog-metadata-inventory\".\"journal\"\nGROUP BY source_ip_address;\n\n#\tsource_ip_address\t_col1\n1\tmy_laptop_IP_address\t12488\n\n# Query S3 Lifecycle expired objects in last 7 days\nSELECT bucket, key, version_id, last_modified_date, record_timestamp\nFROM \"s3tablescatalog/aws-s3\".\"b_aws-news-blog-metadata-inventory\".\"journal\"\nWHERE requester = 's3.amazonaws.com' AND record_type = 'DELETE' AND record_timestamp > (current_date - interval '7' day);\n\n(not applicable to my demo bucket)
\n

The results helped me track the specific objects that were removed, including their timestamps.

\n

Now, I look at the live inventory table:

\n
# Distribution of object tags\nSELECT object_tags, count(object_tags)\nFROM \"s3tablescatalog/aws-s3\".\"b_aws-news-blog-metadata-inventory\".\"inventory\"\nGROUP BY object_tags;\n\n# object_tags    _col1\n1 {Source=Swift} 1\n2 {Source=swift} 1\n3 {}             12486\n\n# Query storage class and size for specific tags\nSELECT storage_class, count(*) as count, sum(size) / 1024 / 1024 as usage\nFROM \"s3tablescatalog/aws-s3\".\"b_aws-news-blog-metadata-inventory\".\"inventory\"\nGROUP BY object_tags['pii=true'], storage_class;\n\n# storage_class count   usage\n1 STANDARD      124884  165\n\n# Find objects with specific user defined metadata\nSELECT key, last_modified_date, user_metadata\nFROM \"s3tablescatalog/aws-s3\".\"b_aws-news-blog-metadata-inventory\".\"inventory\"\nWHERE cardinality(user_metadata) > 0 ORDER BY last_modified_date DESC;\n\n(not applicable to my demo bucket)
\n

These are just a few examples of what is possible with S3 Metadata. Your preferred queries will depend on your use cases. Refer to Analyzing Amazon S3 Metadata with Amazon Athena and Amazon QuickSight in the AWS Storage Blog for more examples.

\n

Pricing and availability
S3 Metadata live inventory and journal tables are available today in US East (N. Virginia), US East (Ohio), and US West (Oregon).

\n

The journal tables are charged $0.30 per million updates. This is a 33 percent drop from our previous price.

\n

For inventory tables, there’s a one-time backfill cost of $0.30 for a million objects to set up the table and generate metadata for existing objects. There are no additional costs if your bucket has less than one billion objects. For buckets with more than a billion objects, there is a monthly fee of $0.10 per million objects per month.

\n

As usual, the Amazon S3 pricing page has all the details.

\n

With S3 Metadata live inventory and journal tables, you can reduce the time and effort required to explore and manage large datasets. You get an up-to-date view of your storage and a record of changes, and both are available as Iceberg tables you can query on demand. You can discover data faster, power compliance workflows, and optimize your ML pipelines.

\n

You can get started by enabling metadata inventory on your S3 bucket through the AWS console, AWS Command Line Interface (AWS CLI), or AWS SDKs. When they’re enabled, the journal and live inventory tables are automatically created and updated. To learn more, visit the S3 Metadata Documentation page.

\n

— seb

\n

Update 7/15/2025: Revised some code and updated Region list.

", "url": "https://aws.amazon.com/blogs/aws/amazon-s3-metadata-now-supports-metadata-for-all-your-s3-objects/", "title": "Amazon S3 Metadata now supports metadata for all your S3 objects", "date_modified": "2025-07-15T23:33:22.000Z" }, { "content_html": "Comments", "url": "https://www.lesswrong.com/posts/dxiConBZTd33sFaRC/field-notes-from-shipping-real-code-with-claude", "title": "Field Notes on Shipping with Claude Code", "date_modified": "2025-07-15T09:01:40.000Z" }, { "content_html": "Comments", "url": "https://www.theverge.com/openai/705999/google-windsurf-ceo-openai", "title": "OpenAI's Windsurf deal is off, and its CEO is going to Google", "date_modified": "2025-07-11T21:35:31.000Z" }, { "content_html": "Comments", "url": "https://simonwillison.net/2025/Jul/10/grok-4/", "title": "Grok 4", "date_modified": "2025-07-10T19:43:44.000Z" }, { "content_html": "Comments", "url": "https://h0x0er.github.io/blog/2025/06/29/ebpf-connecting-with-container-runtimes/", "title": "eBPF: Connecting with Container Runtimes", "date_modified": "2025-07-10T19:10:28.000Z" }, { "content_html": "Comments", "url": "https://github.com/afnanenayet/diffsitter", "title": "Diffsitter – A Tree-sitter based AST difftool to get meaningful semantic diffs", "date_modified": "2025-07-10T12:51:19.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/xai/status/1943158495588815072", "title": "Grok 4 Launch [video]", "date_modified": "2025-07-10T04:02:01.000Z" }, { "content_html": "Comments", "url": "https://comet.perplexity.ai/?a=b", "title": "Perplexity Comet", "date_modified": "2025-07-09T19:14:02.000Z" }, { "content_html": "Comments", "url": "https://www.beam.cloud/blog/agentic-apps", "title": "The Architecture Behind Lovable and Bolt", "date_modified": "2025-07-09T16:10:52.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=44490863", "title": "Launch HN: Morph (YC S23) – Apply AI code edits at 4,500 tokens/sec", "date_modified": "2025-07-07T14:40:45.000Z" }, { "content_html": "Comments", "url": "https://arxiv.org/abs/2506.17298", "title": "Mercury: Ultra-Fast Language Models Based on Diffusion", "date_modified": "2025-07-07T12:31:08.000Z" }, { "content_html": "Comments", "url": "https://github.com/JFryy/systemd-lsp", "title": "Show HN: A Language Server Implementation for SystemD Unit Files", "date_modified": "2025-07-07T00:57:25.000Z" }, { "content_html": "Comments", "url": "https://www.indragie.com/blog/i-shipped-a-macos-app-built-entirely-by-claude-code", "title": "Building a Mac app with Claude code", "date_modified": "2025-07-06T14:55:36.000Z" }, { "content_html": "Comments", "url": "https://9to5mac.com/2025/07/04/apple-just-released-a-weirdly-interesting-coding-language-model/", "title": "Apple just released a weirdly interesting coding language model", "date_modified": "2025-07-05T11:44:05.000Z" }, { "content_html": "

Comments

", "url": "https://pythonspeed.com/articles/different-ways-speed/", "title": "500× faster: Four different ways to speed up your code", "date_modified": "2025-07-02T18:12:32.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=44441530", "title": "Ask HN: Why there is no demand for my SaaS when competition is killing it?", "date_modified": "2025-07-02T09:05:43.000Z" }, { "content_html": "Comments", "url": "https://jamesbvaughan.com/bidirectional-editing/", "title": "Code⇄GUI bidirectional editing via LSP", "date_modified": "2025-07-01T16:43:19.000Z" }, { "content_html": "Comments", "url": "https://branching.app", "title": "Version Control for AI Coding", "date_modified": "2025-07-01T10:00:33.000Z" }, { "content_html": "Comments", "url": "https://docs.anthropic.com/en/docs/claude-code/hooks", "title": "Claude Code now supports Hooks", "date_modified": "2025-07-01T00:01:15.000Z" }, { "content_html": "Comments", "url": "https://docs.anthropic.com/en/docs/claude-code/hooks", "title": "Claude Code now supports hooks", "date_modified": "2025-07-01T00:01:15.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=44426233", "title": "Ask HN: What's the 2025 stack for a self-hosted photo library with local AI?", "date_modified": "2025-06-30T18:10:50.000Z" }, { "content_html": "Comments", "url": "https://tesseral.com/blog/b2b-auth-isnt-that-similar-to-b2c-auth", "title": "Auth for B2B SaaS: it's not like auth for consumer software", "date_modified": "2025-06-30T15:06:14.000Z" }, { "content_html": "Comments", "url": "https://untested.sonnet.io/notes/new-enso-first-public-beta/", "title": "Show HN: New Ensō – first public beta", "date_modified": "2025-06-30T11:02:55.000Z" }, { "content_html": "Comments", "url": "https://not-matthias.github.io/posts/anticheat-update-tracking/", "title": "Anticheat Update Tracking", "date_modified": "2025-06-29T21:06:57.000Z" }, { "content_html": "Comments", "url": "https://www.dolthub.com/blog/2025-06-03-people-keep-inventing-prolly-trees/", "title": "People Keep Inventing Prolly Trees", "date_modified": "2025-06-28T20:01:54.000Z" }, { "content_html": "Comments", "url": "https://hiandrewquinn.github.io/til-site/posts/save-your-disk-write-files-directly-into-ram-with-dev-shm/", "title": "Save your disk, write files directly into RAM with /dev/shm", "date_modified": "2025-06-26T23:04:30.000Z" }, { "content_html": "How we use data lineage to optimize ClickHouse table deployments and avoid unnecessary and expensive data migrations.", "url": "https://www.tinybird.co/blog-posts/when-not-to-migrate-your-data", "title": "How we automatically handle ClickHouse schema migrations", "date_modified": "2025-06-26T22:56:12.656Z" }, { "content_html": "Comments", "url": "https://depot.dev/blog/container-security-at-scale-building-untrusted-images-safely", "title": "Building untrusted container images safely at scale", "date_modified": "2025-06-26T18:22:05.000Z" }, { "content_html": "Comments", "url": "https://developers.googleblog.com/en/introducing-gemma-3n-developer-guide/", "title": "Introducing Gemma 3n", "date_modified": "2025-06-26T17:03:43.000Z" }, { "content_html": "Comments", "url": "https://www.cubic.dev/blog/learnings-from-building-ai-agents", "title": "Learnings from Building AI Agents", "date_modified": "2025-06-26T12:45:04.000Z" }, { "content_html": "

Comments

", "url": "https://technicalwriting.dev/ai/agents/", "title": "Docs for AI agents", "date_modified": "2025-06-25T23:31:41.000Z" }, { "content_html": "Comments", "url": "https://github.com/google-gemini/gemini-cli", "title": "Gemini CLI", "date_modified": "2025-06-25T13:10:46.000Z" }, { "content_html": "Comments", "url": "https://github.com/useautumn/autumn", "title": "Show HN: Autumn – Open-source infra over Stripe", "date_modified": "2025-06-24T12:48:48.000Z" }, { "content_html": "

Comments

", "url": "https://antonz.org/go-json-v2/", "title": "JSON evolution in Go: from v1 to v2", "date_modified": "2025-06-23T13:19:35.000Z" }, { "content_html": "Comments", "url": "https://marketplace.visualstudio.com/items?itemName=anthropic.claude-code", "title": "Claude Code for VSCode", "date_modified": "2025-06-23T08:07:26.000Z" }, { "content_html": "Comments", "url": "https://clickhouse.com/blog/scaling-observability-beyond-100pb-wide-events-replacing-otel", "title": "ClickHouse scales beyond 100 petabytes of logs", "date_modified": "2025-06-21T09:23:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/hatchet-dev/pickaxe", "title": "Show HN: Pickaxe – a TypeScript library for building AI agents", "date_modified": "2025-06-20T16:07:51.000Z" }, { "content_html": "Comments", "url": "https://support.hashicorp.com/hc/en-us/articles/41802449287955-HCP-Vault-Secrets-End-Of-Life", "title": "HCP Vault Secrets End of Life", "date_modified": "2025-06-20T15:47:21.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/phoenix-new-the-remote-ai-runtime/", "title": "Phoenix.new – Remote AI Runtime for Phoenix", "date_modified": "2025-06-20T14:57:04.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=44317242", "title": "Show HN: I'm building an app to replace Overleaf and Notion", "date_modified": "2025-06-19T10:33:44.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=LCEmiRjPEtQ", "title": "Andrej Karpathy: Software in the era of AI [video]", "date_modified": "2025-06-19T00:33:21.000Z" }, { "content_html": "Comments", "url": "https://airpass.tiagoalves.me/", "title": "Airpass – easily overcome WiFi time limits", "date_modified": "2025-06-18T15:29:30.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/engineering/building-effective-agents", "title": "Building Effective AI Agents", "date_modified": "2025-06-17T17:50:05.000Z" }, { "content_html": "Comments", "url": "https://omarabid.com/claude-magic", "title": "Why Claude Code feels like magic?", "date_modified": "2025-06-17T09:53:21.000Z" }, { "content_html": "Comments", "url": "https://coroot.com/blog/opentelemetry-for-go-measuring-the-overhead/", "title": "OpenTelemetry for Go: Measuring overhead costs", "date_modified": "2025-06-16T15:09:17.000Z" }, { "content_html": "Comments", "url": "https://github.com/noghartt/container-compose", "title": "Show HN: Container-compose – A Docker-compose like tool for Apple containers", "date_modified": "2025-06-15T14:53:40.000Z" }, { "content_html": "Comments", "url": "https://substack.com/home/post/p-165651243", "title": "Being a Force Multiplier", "date_modified": "2025-06-12T23:38:24.000Z" }, { "content_html": "Comments", "url": "https://signoz.io/blog/cicd-observability-with-opentelemetry/", "title": "CI/CD Observability with OpenTelemetry Step by Step Guide", "date_modified": "2025-06-11T12:42:50.000Z" }, { "content_html": "

Comments

", "url": "https://diwank.space/field-notes-from-shipping-real-code-with-claude", "title": "Field Notes From Shipping Real Code With Claude", "date_modified": "2025-06-09T02:16:48.000Z" }, { "content_html": "

Comments

", "url": "https://www.maxemitchell.com/writings/i-read-all-of-cloudflares-claude-generated-commits/", "title": "I Read All Of Cloudflare's Claude-Generated Commits", "date_modified": "2025-06-07T12:00:09.000Z" }, { "content_html": "Comments", "url": "https://github.com/possibilities/claude-composer", "title": "Show HN: Claude Composer", "date_modified": "2025-06-05T22:53:21.000Z" }, { "content_html": "

Today, we’re announcing a new publicly available source of API models for Amazon Web Services (AWS). We are now publishing AWS API models on a daily basis to Maven Central and providing open source access to a new repository on GitHub. This repository includes a definitive, up-to-date source of Smithy API models that define AWS public interface definitions and behaviors.

\n

These Smithy models can be used to better understand AWS services and build developer tools like custom SDKs and command line interfaces (CLIs) for connecting to AWS or testing tools for validating your application integrations on AWS.

\n

Since 2018, we have been generating SDK clients and CLI tools using Smithy models. All AWS services are modeled in Smithy to thoroughly document the API contract including operations and behaviors like protocols, authentication, request and response types, and errors.

\n

With this public resource, you can build and test your own applications that can integrate directly with AWS services with confidence such as:

\n \n

Learn about AWS API models
You can browse the AWS service models directly on GitHub by accessing the api-models-aws repository. This repository contains Smithy models with the JSON AST format for all public AWS API services. All Smithy models consist of shapes and traits. Shapes are instances of types and traits are used to add more information to shapes that might be useful for clients, servers, or documentation.

\n

The AWS models repository contains:

\n \n

For example, when you want to define a RunInstances API in Amazon EC2 service, the model uses service type, an entry point of an API that aggregates resources and operations together. The shape referenced by a member is called its target.

\n
com.amazonaws.ec2#AmazonEC2\": {\n      \"type\": \"service\",\n      \"version\": \"2016-11-15\",\n      \"operations\": [\n....\n        {\n          \"target\": \"com.amazonaws.ec2#RunInstances\"\n        },\n....\n\t  ]
\n

The operation type represents the input, output, traits, and possible errors of an API operation. Operation shapes are bound to resource shapes and service shapes. An operation is defined in the IDL using an operation_statement. In the traits, you can find detailed API information such as documentation, examples, and so on.

\n
\"com.amazonaws.ec2#RunInstances\": {\n      \"type\": \"operation\",\n      \"input\": {\n        \"target\": \"com.amazonaws.ec2#RunInstancesRequest\"\n      },\n      \"output\": {\n        \"target\": \"com.amazonaws.ec2#Reservation\"\n      },\n      \"traits\": {\n        \"smithy.api#documentation\": \"<p>Launches the specified number of instances using an AMI for which you have....\",\n        smithy.api#examples\": [\n          {\n            \"title\": \"To launch an instance\",\n            \"documentation\": \"This example launches an instance using the specified AMI, instance type, security group, subnet, block device mapping, and tags.\",\n            \"input\": {\n              \"BlockDeviceMappings\": [\n                {\n                  \"DeviceName\": \"/dev/sdh\",\n                  \"Ebs\": {\n                    \"VolumeSize\": 100\n                  }\n                }\n              ],\n              \"ImageId\": \"ami-abc12345\",\n              \"InstanceType\": \"t2.micro\",\n              \"KeyName\": \"my-key-pair\",\n              \"MaxCount\": 1,\n              \"MinCount\": 1,\n              \"SecurityGroupIds\": [\n                \"sg-1a2b3c4d\"\n              ],\n              \"SubnetId\": \"subnet-6e7f829e\",\n              \"TagSpecifications\": [\n                {\n                  \"ResourceType\": \"instance\",\n                  \"Tags\": [\n                    {\n                      \"Key\": \"Purpose\",\n                      \"Value\": \"test\"\n                    }\n                  ]\n                }\n              ]\n            },\n            \"output\": {}\n          }\n        ]\n      }\n    },
\n

We use Smithy extensively to model our service APIs and provide the daily releases of the AWS SDKs and AWS CLI. AWS API models can be helpful for implementing server stubs to interact with AWS services.

\n

How to build with AWS API models
Smithy API models provide building resources such as build tools, client or server code generators, IDE support, and implementations. For example, with Smithy CLI, you can easily build your models, run ad-hoc validation, compare models for differences, query models, and more. The Smithy CLI makes it easy to get started working with Smithy without setting up Java or using the Smithy Gradle Plugins.

\n

I want to show two examples how to build your own applications with AWS API models and Smithy build tools.

\n \n

Now available
You can now access AWS API models on a daily basis providing open-source access on the AWS API models repository and service model packages available on Maven Central. You can import models and add dependencies using the maven package of your choice.

\n

To learn more about the AWS preferred API modeling language, visit Smithy.io and its code generation guide. To learn more each AWS SDKs, visit Tools to Build on AWS and its respective repository for SDK specific support or through your usual AWS Support contacts.

\n

— Channy

", "url": "https://aws.amazon.com/blogs/aws/introducing-aws-api-models-and-publicly-available-resources-for-aws-api-definitions/", "title": "Introducing AWS API models and publicly available resources for AWS API definitions", "date_modified": "2025-06-05T20:18:26.000Z" }, { "content_html": "Comments", "url": "https://www.nytimes.com/2025/05/21/climate/ai-weather-models-aurora-microsoft.html", "title": "Aurora, a foundation model for the Earth system", "date_modified": "2025-06-05T19:17:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/hyperdxio/hyperdx", "title": "Show HN: ClickStack – Open-source Datadog alternative by ClickHouse and HyperDX", "date_modified": "2025-06-05T18:01:21.000Z" }, { "content_html": "Comments", "url": "https://aavetis.github.io/ai-pr-watcher/", "title": "Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance", "date_modified": "2025-06-05T06:22:09.000Z" }, { "content_html": "Comments", "url": "https://help.openai.com/en/articles/11428266-codex-changelog", "title": "Codex Changelog: agent internet access, voice dictation and update existing PRs", "date_modified": "2025-06-04T06:44:29.000Z" }, { "content_html": "Comments", "url": "https://steipete.me/posts/2025/claude-code-is-my-computer", "title": "Claude Code Is My Computer", "date_modified": "2025-06-03T15:14:52.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/youre-all-nuts/", "title": "My AI skeptic friends are all nuts", "date_modified": "2025-06-02T21:09:53.000Z" }, { "content_html": "Comments", "url": "https://www.wsj.com/articles/snowflake-to-buy-crunchy-data-for-250-million-233543ab", "title": "Snowflake to Buy Crunchy Data for $250M", "date_modified": "2025-06-02T20:01:49.000Z" }, { "content_html": "Comments", "url": "https://github.com/cloudflare/workers-oauth-provider/commits/main/", "title": "Cloudlflare builds OAuth with Claude and publishes all the prompts", "date_modified": "2025-06-02T14:24:54.000Z" }, { "content_html": "Comments", "url": "https://github.com/cloudflare/workers-oauth-provider/", "title": "Cloudlflare builds OAuth with Claude and publishes all the prompts", "date_modified": "2025-06-02T14:24:54.000Z" }, { "content_html": "Comments", "url": "https://github.com/openai/codex/discussions/1174", "title": "Codex CLI is going native", "date_modified": "2025-06-01T11:22:55.000Z" }, { "content_html": "Comments", "url": "https://commaok.xyz/post/simple-backoff/", "title": "Simpler Backoff", "date_modified": "2025-05-31T04:43:14.000Z" }, { "content_html": "Comments", "url": "https://jamesin.substack.com/p/whats-working-for-yc-companies-since", "title": "What's Working for YC Companies Since the AI Boom", "date_modified": "2025-05-30T14:24:36.000Z" }, { "content_html": "Comments", "url": "https://github.com/microsandbox/microsandbox", "title": "Microsandbox: Virtual Machines that feel and perform like containers", "date_modified": "2025-05-30T13:20:04.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/grants-ga", "title": "A new generation of Tailscale access controls", "date_modified": "2025-05-29T18:23:04.000Z" }, { "content_html": "

Cloudflare Workers Builds is our CI/CD product that makes it easy to build and deploy Workers applications every time code is pushed to GitHub or GitLab. What makes Workers Builds special is that projects can be built and deployed with minimal configuration. Just hook up your project and let us take care of the rest!

But what happens when things go wrong, such as failing to install tools or dependencies? What usually happens is that we don’t fix the problem until a customer contacts us about it, at which point many other customers have likely faced the same issue. This can be a frustrating experience for both us and our customers because of the lag time between issues occurring and us fixing them.

We want Workers Builds to be reliable, fast, and easy to use so that developers can focus on building, not dealing with our bugs. That’s why we recently started building an error detection system that can detect, categorize, and surface all build issues occurring on Workers Builds, enabling us to proactively fix issues and add missing features.

It’s also no secret that we’re big fans of being “Customer Zero” at Cloudflare, and Workers Builds is itself a product that’s built end-to-end on our Developer Platform using Workers, Durable Objects, Hyperdrive, Containers, Queues, Workers KV, R2, and Workers Observability.

In this post, we will dive into how we used the Cloudflare Developer Platform to check for issues across more than 1 million Durable Objects.

\n
\n

Background: Workers Builds architecture

\n \n \n \n
\n

Back in October 2024, we wrote about how we built Workers Builds entirely on the Workers platform. To recap, Builds is built using Workers, Durable Objects, Workers KV, R2, Queues, Hyperdrive, and a Postgres database. Some of these things were not present when launched back in October (for example, Queues and KV). But the core of the architecture is the same.

A client Worker receives GitHub/GitLab webhooks and stores build metadata in Postgres (via Hyperdrive). A build management Worker uses two Durable Object classes: a Scheduler class to find builds in Postgres that need scheduling, and a class called BuildBuddy to manage the lifecycle of a build. When a build needs to be started, Scheduler creates a new BuildBuddy instance which is responsible for creating a container for the build (using Cloudflare Containers), monitoring the container with health checks, and receiving build logs so that they can be viewed in the Cloudflare Dashboard.

\n
\n \n

In addition to this core scheduling logic, we have several Workers Queues for background work such as sending PR comments to GitHub/GitLab.

\n
\n

The problem: builds are failing

\n \n \n \n
\n

While this architecture has worked well for us so far, we found ourselves with a problem: compared to Cloudflare Pages, a concerning percentage of builds were failing. We needed to dig deeper and figure out what was wrong, and understand how we could improve Workers Builds so that developers can focus more on shipping instead of build failures.

\n
\n

Types of build failures

\n \n \n \n
\n

Not all build failures are the same. We have several categories of failures that we monitor:

The first few failure types were straightforward, and we’ve been able to track down and fix issues in our build system and control plane to improve what we call “build completion rate”. We define build completion as the following:

  1. We successfully started the build.

  2. We attempted to install tools/dependencies (considering failures as “user error”).

  3. We attempted to run the user-defined build/deploy commands (again, considering failures as “user error”).

  4. We successfully marked the build as stopped in our database.

For example, we had a bug where builds for a deleted Worker would attempt to run and continuously fail, which affected our build completion rate metric.

\n
\n

User error

\n \n \n \n
\n

We’ve made a lot of progress improving the reliability of build and container orchestration, but we had a significant percentage of build failures in the “user error” metric. We started asking ourselves “is this actually user error? Or is there a problem with the product itself?”

This presented a challenge because questions like “did the build command fail due to a bug in the build system, or user error?” are a lot harder to answer than pass/fail issues like failing to create a container for the build. To answer these questions, we had to build something new, something smarter.

\n
\n

Build logs

\n \n \n \n
\n

The most obvious way to determine why a build failed is to look at its logs. When spot-checking build failures, we can typically identify what went wrong. For example, some builds fail to install dependencies because of an out of date lockfile (e.g. package-lock.json out of date with package.json). But looking through build failures one by one doesn’t scale. We didn’t want engineers looking through customer build logs without at least suspecting that there was an issue with our build system that we could fix.

\n
\n

Automating error detection

\n \n \n \n
\n

At this point, next steps were clear: we needed an automated way to identify why a build failed based on build logs, and provide a way for engineers to see what the top issues were while ensuring privacy (e.g. removing account-specific identifiers and file paths from the aggregate data).

\n
\n

Detecting errors in build logs using Workers Queues

\n \n \n \n
\n

The first thing we needed was a way to categorize build errors after a build fails. To do this, we created a queue named BuildErrorsQueue to process builds and look for errors. After a build fails, BuildBuddy will send the build ID to BuildErrorsQueue which fetches the logs, checks for issues, and saves results to Postgres.

\n
\n \n

We started out with a few static patterns to match things like Wrangler errors in log lines:

\n 
export const DetectedErrorCodes = {\n  wrangler_error: {\n    detect: async (lines: LogLines) => {\n      const errors: DetectedError[] = []\n      for (const line of lines) {\n        if (line[2].trim().startsWith('✘ [ERROR]')) {\n          errors.push({\n            error_code: 'wrangler_error',\n            error_group: getWranglerLogGroupFromLogLine(line, wranglerRegexMatchers),\n            detected_on: new Date(),\n            lines_matched: [line],\n          })\n        }\n      }\n      return errors\n    },\n  },\n  installing_tools_or_dependencies_failed: { ... },\n}
\n

It wouldn’t be useful if all Wrangler errors were grouped under a single generic “wrangler_error” code, so we further grouped them by normalizing the log lines into groups:

\n 
function getWranglerLogGroupFromLogLine(\n  logLine: LogLine,\n  regexMatchers: RegexMatcher[]\n): string {\n  const original = logLine[2].trim().replaceAll(/[\\t\\n\\r]+/g, ' ')\n  let message = original\n  let group = original\n  for (const { mustMatch, patterns, stopOnMatch, name, useNameAsGroup } of regexMatchers) {\n    if (mustMatch !== undefined) {\n      const matched = matchLineToRegexes(message, mustMatch)\n      if (!matched) continue\n    }\n    if (patterns) {\n      for (const [pattern, mask] of patterns) {\n        message = message.replaceAll(pattern, mask)\n      }\n    }\n    if (useNameAsGroup === true) {\n      group = name\n    } else {\n      group = message\n    }\n    if (Boolean(stopOnMatch) && message !== original) break\n  }\n  return group\n}\n\nconst wranglerRegexMatchers: RegexMatcher[] = [\n  {\n    name: 'could_not_resolve',\n    // ✘ [ERROR] Could not resolve \"./balance\"\n    // ✘ [ERROR] Could not resolve \"node:string_decoder\" (originally \"string_decoder/\")\n    mustMatch: [/^✘ \\[ERROR\\] Could not resolve \"[@\\w :/\\\\.-]*\"/i],\n    stopOnMatch: true,\n    patterns: [\n      [/(?<=^✘ \\[ERROR\\] Could not resolve \")[@\\w :/\\\\.-]*(?=\")/gi, '<MODULE>'],\n      [/(?<=\\(originally \")[@\\w :/\\\\.-]*(?=\")/gi, '<MODULE>'],\n    ],\n  },\n  {\n    name: 'no_matching_export_for_import',\n    // ✘ [ERROR] No matching export in \"src/db/schemas/index.ts\" for import \"someCoolTable\"\n    mustMatch: [/^✘ \\[ERROR\\] No matching export in \"/i],\n    stopOnMatch: true,\n    patterns: [\n      [/(?<=^✘ \\[ERROR\\] No matching export in \")[@~\\w:/\\\\.-]*(?=\")/gi, '<MODULE>'],\n      [/(?<=\" for import \")[\\w-]*(?=\")/gi, '<IMPORT>'],\n    ],\n  },\n  // ...many more added over time\n]
\n

Once we had our error detection matchers and normalizing logic in place, implementing the BuildErrorsQueue consumer was easy:

\n 
export async function handleQueue(\n  batch: MessageBatch,\n  env: Bindings,\n  ctx: ExecutionContext\n): Promise<void> {\n  ...\n  await pMap(batch.messages, async (msg) => {\n    try {\n      const { build_id } = BuildErrorsQueueMessageBody.parse(msg.body)\n      await store.buildErrors.deleteErrorsByBuildId({ build_id })\n      const bb = getBuildBuddy(env, build_id)\n      const errors: DetectedError[] = []\n      let cursor: LogsCursor | undefined\n      let hasMore = false\n\n      do {\n        using maybeNewLogs = await bb.getLogs(cursor, false)\n        const newLogs = LogsWithCursor.parse(maybeNewLogs)\n        cursor = newLogs.cursor\n        const newErrors = await detectErrorsInLogLines(newLogs.lines)\n        errors.push(...newErrors)\n        hasMore = Boolean(cursor) && newLogs.lines.length > 0\n      } while (hasMore)\n\n      if (errors.length > 0) {\n        await store.buildErrors.insertErrors(\n          errors.map((e) => ({\n            build_id,\n            error_code: e.error_code,\n            error_group: e.error_group,\n          }))\n        )\n      }\n      msg.ack()\n    } catch (e) {\n      msg.retry()\n      sentry.captureException(e)\n    }\n  })\n}
\n

Here, we’re fetching logs from each build’s BuildBuddy Durable Object, detecting why it failed using the matchers we wrote, and saving errors to the Postgres DB. We also delete any existing errors for when we improve our error detection patterns to prevent subsequent runs from adding duplicate data to our database.

\n
\n

What about historical builds?

\n \n \n \n
\n

The BuildErrorsQueue was great for new builds, but this meant we still didn’t know why all the previous build failures happened other than “user error”. We considered only tracking errors in new builds, but this was unacceptable because it would significantly slow down our ability to improve our error detection system because each iteration would require us to wait days to identify issues we need to prioritize.

\n
\n

Problem: logs are stored across one million+ Durable Objects

\n \n \n \n
\n

Remember how every build has an associated BuildBuddy DO to store logs? This is a great design for ensuring our logging pipeline scales with our customers, but it presented a challenge when trying to aggregate issues based on logs because something would need to go through all historical builds (>1 million at the time) to fetch logs and detect why they failed.

If we were using Go and Kubernetes, we might solve this using a long-running container that goes through all builds and runs our error detection. But how do we solve this in Workers?

\n
\n

How do we backfill errors for historical builds?

\n \n \n \n
\n

At this point, we already had the Queue to process new builds. If we could somehow send all of the old build IDs to the queue, it could scan them all quickly using Queues concurrent consumers to quickly work through all builds. We thought about hacking together a local script to fetch all of the log IDs and sending them to an API to put them on a queue. But we wanted something more secure and easier to use so that running a new backfill was as simple as an API call.

That’s when an idea hit us: what if we used a Durable Object with alarms to fetch a range of builds and send them to BuildErrorsQueue? At first, it seemed far-fetched, given that Durable Object alarms have a limited amount of work they can do per invocation. But wait, if AI Agents built on Durable Objects can manage background tasks, why can’t we fetch millions of build IDs and forward them to queues?

\n
\n

Building a Build Errors Agent with Durable Objects

\n \n \n \n
\n

The idea was simple: create a Durable Object class named BuildErrorsAgent and run a single instance that loops through the specified range of builds in the database and sends them to BuildErrorsQueue.

\n
\n \n

The first thing we did was set up an RPC method to start a backfill and save the parameters in Durable Object KV storage so that it can be read each time the alarm executes:

\n 
async start({\n  min_build_id,\n  max_build_id,\n}: {\n  min_build_id: BuildRecord['build_id']\n  max_build_id: BuildRecord['build_id']\n}): Promise<void> {\n  logger.setTags({ handler: 'start', environment: this.env.ENVIRONMENT })\n  try {\n    if (min_build_id < 0) throw new Error('min_build_id cannot be negative')\n    if (max_build_id < min_build_id) {\n      throw new Error('max_build_id cannot be less than min_build_id')\n    }\n    const [started_on, stopped_on] = await Promise.all([\n      this.kv.get('started_on'),\n      this.kv.get('stopped_on'),\n    ])\n    await match({ started_on, stopped_on })\n      .with({ started_on: P.not(null), stopped_on: P.nullish }, () => {\n        throw new Error('BuildErrorsAgent is already running')\n      })\n      .otherwise(async () => {\n        // delete all existing data and start queueing failed builds\n        await this.state.storage.deleteAlarm()\n        await this.state.storage.deleteAll()\n        this.kv.put('started_on', new Date())\n        this.kv.put('config', { min_build_id, max_build_id })\n        void this.state.storage.setAlarm(this.getNextAlarmDate())\n      })\n  } catch (e) {\n    this.sentry.captureException(e)\n    throw e\n  }\n}
\n

The most important part of the implementation is the alarm that runs every second until the job is complete. Each alarm invocation has the following steps:

  1. Set a new alarm (always first to ensure an error doesn’t cause it to stop).

  2. Retrieve state from KV.

  3. Validate that the agent is supposed to be running:

    1. Ensure the agent is supposed to be running.

    2. Ensure we haven’t reached the max build ID set in the config.

  4. Finally, queue up another batch of builds by querying Postgres and sending to the BuildErrorsQueue.

\n
\n \n
\n 
async alarm(): Promise<void> {\n  logger.setTags({ handler: 'alarm', environment: this.env.ENVIRONMENT })\n  try {\n    void this.state.storage.setAlarm(Date.now() + 1000)\n    const kvState = await this.getKVState()\n    this.sentry.setContext('BuildErrorsAgent', kvState)\n    const ctxLogger = logger.withFields({ state: JSON.stringify(kvState) })\n\n    await match(kvState)\n      .with({ started_on: P.nullish }, async () => {\n        ctxLogger.info('BuildErrorsAgent is not started, cancelling alarm')\n        await this.state.storage.deleteAlarm()\n      })\n      .with({ stopped_on: P.not(null) }, async () => {\n        ctxLogger.info('BuildErrorsAgent is stopped, cancelling alarm')\n        await this.state.storage.deleteAlarm()\n      })\n      .with(\n        // we should never have started_on set without config set, but just in case\n        { started_on: P.not(null), config: P.nullish },\n        async () => {\n          const msg =\n            'BuildErrorsAgent started but config is empty, stopping and cancelling alarm'\n          ctxLogger.error(msg)\n          this.sentry.captureException(new Error(msg))\n          this.kv.put('stopped_on', new Date())\n          await this.state.storage.deleteAlarm()\n        }\n      )\n      .when(\n        // make sure there are still builds to enqueue\n        (s) =>\n          s.latest_build_id !== null &&\n          s.config !== null &&\n          s.latest_build_id >= s.config.max_build_id,\n        async () => {\n          ctxLogger.info('BuildErrorsAgent job complete, cancelling alarm')\n          this.kv.put('stopped_on', new Date())\n          await this.state.storage.deleteAlarm()\n        }\n      )\n      .with(\n        {\n          started_on: P.not(null),\n          stopped_on: P.nullish,\n          config: P.not(null),\n          latest_build_id: P.any,\n        },\n        async ({ config, latest_build_id }) => {\n          // 1. select batch of ~1000 builds\n          // 2. send them to Queues 100 at a time, updating\n          //    latest_build_id after each batch is sent\n          const failedBuilds = await this.store.builds.selectFailedBuilds({\n            min_build_id: latest_build_id !== null ? latest_build_id + 1 : config.min_build_id,\n            max_build_id: config.max_build_id,\n            limit: 1000,\n          })\n          if (failedBuilds.length === 0) {\n            ctxLogger.info(`BuildErrorsAgent: ran out of builds, stopping and cancelling alarm`)\n            this.kv.put('stopped_on', new Date())\n            await this.state.storage.deleteAlarm()\n          }\n\n          for (\n            let i = 0;\n            i < BUILDS_PER_ALARM_RUN && i < failedBuilds.length;\n            i += QUEUES_BATCH_SIZE\n          ) {\n            const batch = failedBuilds\n              .slice(i, QUEUES_BATCH_SIZE)\n              .map((build) => ({ body: build }))\n\n            if (batch.length === 0) {\n              ctxLogger.info(`BuildErrorsAgent: ran out of builds in current batch`)\n              break\n            }\n            ctxLogger.info(\n              `BuildErrorsAgent: sending ${batch.length} builds to build errors queue`\n            )\n            await this.env.BUILD_ERRORS_QUEUE.sendBatch(batch)\n            this.kv.put(\n              'latest_build_id',\n              Math.max(...batch.map((m) => m.body.build_id).concat(latest_build_id ?? 0))\n            )\n\n            this.kv.put(\n              'total_builds_processed',\n              ((await this.kv.get('total_builds_processed')) ?? 0) + batch.length\n            )\n          }\n        }\n      )\n      .otherwise(() => {\n        const msg = 'BuildErrorsAgent has nothing to do - this should never happen'\n        this.sentry.captureException(msg)\n        ctxLogger.info(msg)\n      })\n  } catch (e) {\n    this.sentry.captureException(e)\n    throw e\n  }\n}
\n

Using pattern matching with ts-pattern made it much easier to understand what states we were expecting and what will happen compared to procedural code. We considered using a more powerful library like XState, but decided on ts-pattern due to its simplicity.

\n
\n

Running the backfill

\n \n \n \n
\n

Once everything rolled out, we were able to trigger an errors backfill for over a million failed builds in a couple of hours with a single API call, categorizing 80% of failed builds on the first run. With a fast backfill process, we were able to iterate on our regex matchers to further refine our error detection and improve error grouping. Here’s what the error list looks like in our staging environment:

\n
\n \n
\n
\n

Fixes and improvements

\n \n \n \n
\n

Having a better understanding of what’s going wrong has already enabled us to make several improvements:

We’re still working on fixing other bugs we’ve found, but we’re making steady progress. Reliability is a feature we’re striving for in Workers Builds, and this project has helped us make meaningful progress towards that goal. Instead of waiting for people to contact support for issues, we’re able to proactively identify and fix issues (and catch regressions more easily).

One of the great things about building on the Developer Platform is how easy it is to ship things. The core of this error detection pipeline (the Queue and Durable Object) only took two days to build, which meant we could spend more time working on improving Workers Builds instead of spending weeks on the error detection pipeline itself.

\n
\n

What’s next?

\n \n \n \n
\n

In addition to continuing to improve build reliability and speed, we’ve also started thinking about other ways to help developers build their applications on Workers. For example, we built a Builds MCP server that allows users to debug builds directly in Cursor/Claude/etc. We’re also thinking about ways we can expose these detected issues in the Cloudflare Dashboard so that users can identify issues more easily without scrolling through hundreds of logs.

\n
\n

Ready to get started?

\n \n \n \n
\n

Building applications on Workers has never been easier! Try deploying a Durable Object-backed chat application with Workers Builds: 

", "url": "https://blog.cloudflare.com/detecting-workers-builds-errors-across-1-million-durable-durable-objects/", "title": "Let’s DO this: detecting Workers Builds errors across 1 million Durable Objects", "date_modified": "2025-05-29T13:00:00.000Z" }, { "content_html": "

Comments

", "url": "https://clickhouse.com/blog/building-a-distributed-cache-for-s3", "title": "Building a Distributed Cache for S3", "date_modified": "2025-05-29T06:52:40.000Z" }, { "content_html": "Comments", "url": "https://github.com/wzhudev/reverse-linear-sync-engine", "title": "Reverse engineering of Linear's sync engine", "date_modified": "2025-05-29T04:29:30.000Z" }, { "content_html": "

Yeah I know this place is generally super anti-AI. But I figured it’s dishonest to not also post it here. I’d love to see more nuanced posts on this topic here.

\n

Comments

", "url": "https://steveklabnik.com/writing/i-am-disappointed-in-the-ai-discourse/", "title": "I am disappointed in the AI discourse", "date_modified": "2025-05-28T17:35:48.000Z" }, { "content_html": "Comments", "url": "https://ducklake.select/", "title": "DuckLake is an integrated data lake and catalog format", "date_modified": "2025-05-27T13:43:11.000Z" }, { "content_html": "Comments", "url": "https://www.allthingsdistributed.com/2025/05/just-make-it-scale-an-aurora-dsql-story.html", "title": "Just make it scale: An Aurora DSQL story", "date_modified": "2025-05-27T11:31:02.000Z" }, { "content_html": "Comments", "url": "https://maych.in/blog/its-time-to-give-nix-a-chance/", "title": "I think it's time to give Nix a chance", "date_modified": "2025-05-26T15:56:15.000Z" }, { "content_html": "Comments", "url": "https://www.aluxian.com/claude-code-does-our-releases-now/", "title": "Claude Code does our releases now", "date_modified": "2025-05-26T03:22:14.000Z" }, { "content_html": "Comments", "url": "https://chrlschn.dev/blog/2025/05/beware-the-complexity-merchants/", "title": "Beware the Complexity Merchants", "date_modified": "2025-05-25T19:25:58.000Z" }, { "content_html": "Comments", "url": "https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/", "title": "I used o3 to find a remote zeroday in the Linux SMB implementation", "date_modified": "2025-05-24T14:25:45.000Z" }, { "content_html": "Comments", "url": "https://www.wordonfire.org/articles/remembering-alasdair-macintyre-1929-2025/", "title": "Alasdair MacIntyre Has Died", "date_modified": "2025-05-23T11:37:07.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/claude-4", "title": "Claude 4", "date_modified": "2025-05-22T16:34:42.000Z" }, { "content_html": "Comments", "url": "https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/", "title": "How I used o3 to find a remote 0-day vulnerability in the Linux kernel (ksmbd)", "date_modified": "2025-05-22T10:44:48.000Z" }, { "content_html": "

Comments

", "url": "https://encore.dev/blog/mcp-server", "title": "Encore's MCP Server enables your AI tools to introspect your application", "date_modified": "2025-05-22T07:56:32.000Z" }, { "content_html": "

I’m pleased to announce developers can now programmatically disable Apple System Integrity Protection (SIP) on their Amazon EC2 Mac instances. System Integrity Protection (SIP), also known as rootless, is a security feature introduced by Apple in OS X El Capitan (2015, version 10.11). It’s designed to protect the system from potentially harmful software by restricting the power of the root user account. SIP is enabled by default on macOS.

\n

SIP safeguards the system by preventing modification of protected files and folders, restricting access to system-owned files and directories, and blocking unauthorized software from selecting a startup disk. The primary goal of SIP is to address the security risk linked to unrestricted root access, which could potentially allow malware to gain full control of a device with just one password or vulnerability. By implementing this protection, Apple aims to ensure a higher level of security for macOS users, especially considering that many users operate on administrative accounts with weak or no passwords.

\n

While SIP provides excellent protection against malware for everyday use, developers might occasionally need to temporarily disable it for development and testing purposes. For instance, when creating a new device driver or system extension, disabling SIP is necessary to install and test the code. Additionally, SIP might block access to certain system settings required for your software to function properly. Temporarily disabling SIP grants you the necessary permissions to fine-tune programs for macOS. However, it’s crucial to remember that this is akin to briefly disabling the vault door for authorized maintenance, not leaving it permanently open.

\n

Disabling SIP on a Mac requires physical access to the machine. You have to restart the machine in recovery mode, then disable SIP with the csrutil command line tool, then restart the machine again.

\n

Until today, you had to operate with the standard SIP settings on EC2 Mac instances. The physical access requirement and the need to boot in recovery mode made integrating SIP with the Amazon EC2 control plane and EC2 API challenging. But that’s no longer the case! You can now disable and re-enable SIP at will on your Amazon EC2 Mac instances. Let me show you how.

\n

Let’s see how it works
Imagine I have an Amazon EC2 Mac instance started. It’s a mac2-m2.metal instance, running on an Apple silicon M2 processor. Disabling or enabling SIP is as straightforward as calling a new EC2 API: CreateMacSystemIntegrityProtectionModificationTask. This API is asynchronous; it starts the process of changing the SIP status on your instance. You can monitor progress using another new EC2 API: DescribeMacModificationTasks. All I need to know is the instance ID of the machine I want to work with.

\n

Prerequisites
On Apple silicon based EC2 Mac instances and more recent type of machines, before calling the new EC2 API, I must set the ec2-user user password and enable secure token for that user on macOS. This requires connecting to the machine and typing two commands in the terminal.

\n
# on the target EC2 Mac instance\n# Set a password for the ec2-user user\n~ % sudo /usr/bin/dscl . -passwd /Users/ec2-user\nNew Password: (MyNewPassw0rd)\n\n# Enable secure token, with the same password, for the ec2-user\n# old password is the one you just set with dscl\n~ % sysadminctl -newPassword MyNewPassw0rd -oldPassword MyNewPassw0rd\n2025-03-05 13:16:57.261 sysadminctl[3993:3033024] Attempting to change password for ec2-user…\n2025-03-05 13:16:58.690 sysadminctl[3993:3033024] SecKeychainCopyLogin returned -25294\n2025-03-05 13:16:58.690 sysadminctl[3993:3033024] Failed to update keychain password (-25294)\n2025-03-05 13:16:58.690 sysadminctl[3993:3033024] - Done\n\n# The error about the KeyChain is expected. I never connected with the GUI on this machine, so the Login keychain does not exist\n# you can ignore this error.  The command below shows the list of keychains active in this session\n~ % security list\n    \"/Library/Keychains/System.keychain\"\n\n# Verify that the secure token is ENABLED\n~ % sysadminctl -secureTokenStatus ec2-user\n2025-03-05 13:18:12.456 sysadminctl[4017:3033614] Secure token is ENABLED for user ec2-user\n
\n

Change the SIP status
I don’t need to connect to the machine to toggle the SIP status. I only need to know its instance ID. I open a terminal on my laptop and use the AWS Command Line Interface (AWS CLI) to retrieve the Amazon EC2 Mac instance ID.

\n
 aws ec2 describe-instances \\\n         --query \"Reservations[].Instances[?InstanceType == 'mac2-m2.metal' ].InstanceId\" \\\n         --output text\n\ni-012a5de8da47bdff7
\n

Now, still from the terminal on my laptop, I disable SIP with the create-mac-system-integrity-protection-modification-task command:

\n
echo '{\"rootVolumeUsername\":\"ec2-user\",\"rootVolumePassword\":\"MyNewPassw0rd\"}' > tmpCredentials\naws ec2 create-mac-system-integrity-protection-modification-task \\\n--instance-id \"i-012a5de8da47bdff7\" \\\n--mac-credentials file://./tmpCredentials \\\n--mac-system-integrity-protection-status \"disabled\" && rm tmpCredentials\n\n{\n    \"macModificationTask\": {\n        \"instanceId\": \"i-012a5de8da47bdff7\",\n        \"macModificationTaskId\": \"macmodification-06a4bb89b394ac6d6\",\n        \"macSystemIntegrityProtectionConfig\": {},\n        \"startTime\": \"2025-03-14T14:15:06Z\",\n        \"taskState\": \"pending\",\n        \"taskType\": \"sip-modification\"\n    }\n}
\n

After the task is started, I can check its status with the aws ec2 describe-mac-modification-tasks command.

\n
{\n    \"macModificationTasks\": [\n        {\n            \"instanceId\": \"i-012a5de8da47bdff7\",\n            \"macModificationTaskId\": \"macmodification-06a4bb89b394ac6d6\",\n            \"macSystemIntegrityProtectionConfig\": {\n                \"debuggingRestrictions\": \"\",\n                \"dTraceRestrictions\": \"\",\n                \"filesystemProtections\": \"\",\n                \"kextSigning\": \"\",\n                \"nvramProtections\": \"\",\n                \"status\": \"disabled\"\n            },\n            \"startTime\": \"2025-03-14T14:15:06Z\",\n            \"tags\": [],\n            \"taskState\": \"in-progress\",\n            \"taskType\": \"sip-modification\"\n        },\n...
\n

The instance initiates the process and a series of reboots, during which it becomes unreachable. This process can take 60–90 minutes to complete. After that, when I see the status in the console becoming available again, I connect to the machine through SSH or EC2 Instance Connect, as usual.

\n
➜  ~ ssh ec2-user@54.99.9.99\nWarning: Permanently added '54.99.9.99' (ED25519) to the list of known hosts.\nLast login: Mon Feb 26 08:52:42 2024 from 1.1.1.1\n\n    ┌───┬──┐   __|  __|_  )\n    │ ╷╭╯╷ │   _|  (     /\n    │  └╮  │  ___|\\___|___|\n    │ ╰─┼╯ │  Amazon EC2\n    └───┴──┘  macOS Sonoma 14.3.1\n\n➜  ~ uname -a\nDarwin Mac-mini.local 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:27 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T8103 arm64\n\n➜ ~ csrutil --status \nSystem Integrity Protection status: disabled.\n
\n

When to disable SIP
Disabling SIP should be approached with caution because it opens up the system to potential security risks. However, as I mentioned in the introduction of this post, you might need to disable SIP when developing device drivers or kernel extensions for macOS. Some older applications might also not function correctly when SIP is enabled.

\n

Disabling SIP is also required to turn off Spotlight indexing. Spotlight can help you quickly find apps, documents, emails and other items on your Mac. It’s very convenient on desktop machines, but not so much on a server. When there is no need to index your documents as they change, turning off Spotlight will release some CPU cycles and disk I/O.

\n

Things to know
There are a couple of additional things to know about disabling SIP on Amazon EC2 Mac:

\n \n

These new APIs are available in all Regions where Amazon EC2 Mac is available, at no additional cost. Try them today.

\n— seb", "url": "https://aws.amazon.com/blogs/aws/configure-system-integrity-protection-sip-on-amazon-ec2-mac-instances/", "title": "Configure System Integrity Protection (SIP) on Amazon EC2 Mac instances", "date_modified": "2025-05-21T17:36:31.000Z" }, { "content_html": "Comments", "url": "https://github.com/lastmile-ai/mcp-agent/tree/main/examples/mcp_agent_server", "title": "Show HN: Representing Agents as MCP Servers", "date_modified": "2025-05-21T17:19:52.000Z" }, { "content_html": "Comments", "url": "https://jngiam.bearblog.dev/mcp-large-data/", "title": "LLM function calls don't scale; code orchestration is simpler, more effective", "date_modified": "2025-05-21T17:18:52.000Z" }, { "content_html": "

Comments

", "url": "https://seiya.me/blog/hypervisor-as-a-library", "title": "Hypervisor as a Library", "date_modified": "2025-05-20T15:17:44.000Z" }, { "content_html": "Comments", "url": "https://zackproser.com/blog/openai-codex-review", "title": "OpenAI Codex Review", "date_modified": "2025-05-20T14:29:00.000Z" }, { "content_html": "Comments", "url": "https://zackproser.com/blog/openai-codex-review", "title": "OpenAI Codex hands-on review", "date_modified": "2025-05-20T14:29:00.000Z" }, { "content_html": "Comments", "url": "https://seiya.me/blog/hypervisor-as-a-library", "title": "Hypervisor as a Library", "date_modified": "2025-05-20T06:07:19.000Z" }, { "content_html": "Comments", "url": "https://en.wikipedia.org/wiki/Cleo_(mathematician)", "title": "Cleo, the mathematician that tricked Stack Exchange", "date_modified": "2025-05-20T05:47:55.000Z" }, { "content_html": "Comments", "url": "https://cloudcoding.ai/", "title": "Show HN: Claude Code in the Cloud", "date_modified": "2025-05-19T23:02:38.000Z" }, { "content_html": "Comments", "url": "https://jules.google/", "title": "Jules: An Asynchronous Coding Agent", "date_modified": "2025-05-19T21:12:47.000Z" }, { "content_html": "Comments", "url": "https://docs.anthropic.com/en/docs/claude-code/sdk", "title": "Claude Code SDK – Anthropic", "date_modified": "2025-05-19T18:04:06.000Z" }, { "content_html": "Comments", "url": "https://docs.anthropic.com/en/docs/claude-code/sdk", "title": "Claude Code SDK", "date_modified": "2025-05-19T18:04:06.000Z" }, { "content_html": "Comments", "url": "https://github.blog/changelog/2025-05-19-github-copilot-coding-agent-in-public-preview/", "title": "GitHub Copilot Coding Agent", "date_modified": "2025-05-19T16:17:56.000Z" }, { "content_html": "Comments", "url": "https://github.com/crhuber/kelp", "title": "Kelp – simple replacement for homebrew on macOS", "date_modified": "2025-05-19T14:55:53.000Z" }, { "content_html": "Comments", "url": "https://airport.query.farm/", "title": "Airport for DuckDB", "date_modified": "2025-05-19T11:25:32.000Z" }, { "content_html": "Comments", "url": "https://nghiant3223.github.io/2025/04/15/go-scheduler.html", "title": "Understanding the Go Scheduler", "date_modified": "2025-05-18T17:03:55.000Z" }, { "content_html": "

Comments

", "url": "https://matklad.github.io/2023/11/15/push-ifs-up-and-fors-down.html", "title": "Push Ifs Up And Fors Down (2023)", "date_modified": "2025-05-17T20:08:29.000Z" }, { "content_html": "Comments", "url": "https://martincapodici.com/2025/05/13/production-tests-a-guidebook-for-better-systems-and-more-sleep/", "title": "Production tests: a guidebook for better systems and more sleep", "date_modified": "2025-05-17T09:13:16.000Z" }, { "content_html": "Comments", "url": "https://openai.com/index/introducing-codex/", "title": "A Research Preview of Codex", "date_modified": "2025-05-16T15:02:02.000Z" }, { "content_html": "Comments", "url": "https://xata.io/blog/xata-postgres-with-data-branching-and-pii-anonymization", "title": "Postgres with data branching and PII anonymization", "date_modified": "2025-05-15T08:14:29.000Z" }, { "content_html": "

Comments

", "url": "https://s2.dev/blog/timestamping", "title": "Keeping time on a stream", "date_modified": "2025-05-14T18:11:43.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/4via6-connectivity-to-edge-devices", "title": "Tailscale 4via6 – Connect Edge Deployments at Scale", "date_modified": "2025-05-12T14:00:32.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=43959710", "title": "Ask HN: Cursor or Windsurf?", "date_modified": "2025-05-12T04:41:50.000Z" }, { "content_html": "Comments", "url": "https://warpstreamlabs.github.io/bento/", "title": "Bento gets a makeover", "date_modified": "2025-05-08T21:30:41.000Z" }, { "content_html": "Comments", "url": "https://support.anthropic.com/en/articles/11145838-using-claude-code-with-your-max-plan", "title": "A flat pricing subscription for Claude Code", "date_modified": "2025-05-08T21:12:32.000Z" }, { "content_html": "Comments", "url": "https://github.com/voideditor/void", "title": "Void: Open-source Cursor alternative", "date_modified": "2025-05-08T16:35:34.000Z" }, { "content_html": "Comments", "url": "https://ghiculescu.substack.com/p/nobody-codes-here-anymore", "title": "Notes on rolling out Cursor and Claude Code", "date_modified": "2025-05-08T16:34:39.000Z" }, { "content_html": "

Today we are announcing the general availability of Amazon Elastic Block Store (Amazon EBS) Provisioned Rate for Volume Initialization, a feature that accelerates the transfer of data from an EBS snapshot, a highly durable backup of volumes stored in Amazon Simple Storage Service (Amazon S3) to a new EBS volume.

\n

With Amazon EBS Provisioned Rate for Volume Initialization, you can create fully performant EBS volumes within a predictable amount of time. You can use this feature to speed up the initialization of hundreds of concurrent volumes and instances. You can also use this feature when you need to recover from an existing EBS Snapshot and need your EBS volume to be created and initialized as quickly as possible. You can use this feature to quickly create copies of EBS volumes with EBS Snapshots in a different Availability Zone, AWS Region, or AWS account. Provisioned Rate for Volume Initialization for each volume is charged based on the full snapshot size and the specified volume initialization rate.

\n

This new feature expedites the volume initialization process by fetching the data from an EBS Snapshot to an EBS volume at a consistent rate that you specify between 100 MiB/s and 300 MiB/s. You can specify this volume initialization rate at which the snapshot blocks are to be downloaded from Amazon S3 to the volume.

\n

With specifying the volume initialization rate, you can create a fully performant volume in a predictable time, enabling increased operational efficiency and visibility on the expected time of completion. If you run utilities like fio/dd to expedite volume initialization for your workflows like application recovery and volume copy for testing and development, it will remove the operational burden of managing such scripts with the consistency and predictability to your workflows.

\n

Get started with specifying the volume initialization rate
To get started, you can choose the volume initialization rate when you launch your EC2 instance or create your volume from the snapshot.

\n

1. Create a volume in the EC2 launch wizard
When launching new EC2 instances in the launch wizard of EC2 console, you can enter a desired Volume initialization rate in the Storage (volumes) section.

\n

\n

You can also set the volume initialization rate when creating and modifying the EC2 Launch Templates.

\n

\n

In the AWS Command Line Interface (AWS CLI), you can add VolumeInitializationRate parameter to the block device mappings when you call run-instances command.

\n
aws ec2 run-instances \\\n    --image-id ami-0abcdef1234567890 \\\n    --instance-type t2.micro \\\n    --subnet-id subnet-08fc749671b2d077c \\\n    --security-group-ids sg-0b0384b66d7d692f9 \\\n    --key-name MyKeyPair \\\n    --block-device-mappings file://mapping.json
\n

Contents of mapping.json. This example adds /dev/sdh an empty EBS volume with a size of 8 GiB.

\n
[\n    {\n        \"DeviceName\": \"/dev/sdh\",\n        \"Ebs\": {\n            \"VolumeSize\": 8,\n            \"VolumeType\": \"gp3\",            \n            \"VolumeInitializationRate\": 300\n\t\t } \n     } \n]
\n

To learn more, visit block device mapping options, which defines the EBS volumes and instance store volumes to attach to the instance at launch.

\n

2. Create a volume from snapshots
When you create a volume from snapshots, you can also choose Create volume in the EC2 console and specify the Volume initialization rate.

\n

\n

Confirm your new volume with the initialization rate.

\n

\n

In the AWS CLI, you can use VolumeInitializationRate parameter and when calling create-volume command.

\n
aws ec2 create-volume --region us-east-1 --cli-input-json '{\n    \"AvailabilityZone\": \"us-east-1a\",\n    \"VolumeType\": \"gp3\",\n    \"SnapshotId\": \"snap-07f411eed12ef613a\",\n    \"VolumeInitializationRate\": 300\n}'
\n

If the command is run successfully, you will receive the result below.

\n
{\n    \"AvailabilityZone\": \"us-east-1a\",\n    \"CreateTime\": \"2025-01-03T21:44:53.000Z\",\n    \"Encrypted\": false,\n    \"Size\": 100,\n    \"SnapshotId\": \"snap-07f411eed12ef613a\",\n    \"State\": \"creating\",\n    \"VolumeId\": \"vol-0ba4ed2a280fab5f9\",\n    \"Iops\": 300,\n    \"Tags\": [],\n    \"VolumeType\": \"gp2\",\n    \"MultiAttachEnabled\": false,\n    \"VolumeInitializationRate\": 300\n}
\n

You can also set the volume initialization rate when replacing root volumes of EC2 instances and provisioning EBS volumes using the EBS Container Storage Interface (CSI) driver.

\n

After creation of the volume, EBS will keep track of the hydration progress and publish an Amazon EventBridge notification for EBS to your account when the hydration completes so that they can be certain when their volume is fully performant.

\n

To learn more, visit Create an Amazon EBS volume and Initialize Amazon EBS volumes in the Amazon EBS User Guide.

\n

Now available
Amazon EBS Provisioned Rate for Volume Initialization is now available and supported for all EBS volume types today. You will be charged based on the full snapshot size and the specified volume initialization rate. To learn more, visit Amazon EBS Pricing page.

\n

To learn more about Amazon EBS including this feature, take the free digital course on the AWS Skill Builder portal. Course includes use cases, architecture diagrams and demos.

\n

Give this feature a try in the Amazon EC2 console today and send feedback to AWS re:Post for Amazon EBS or through your usual AWS Support contacts.

\n

— Channy

\n
\n

How is the News Blog doing? Take this 1 minute survey!

\n

(This survey is hosted by an external company. AWS handles your information as described in the AWS Privacy Notice. AWS will own the data gathered via this survey and will not share the information collected with survey respondents.)

", "url": "https://aws.amazon.com/blogs/aws/accelerate-the-transfer-of-data-from-an-amazon-ebs-snapshot-to-a-new-ebs-volume/", "title": "Accelerate the transfer of data from an Amazon EBS snapshot to a new EBS volume", "date_modified": "2025-05-06T21:52:26.000Z" }, { "content_html": "Comments", "url": "https://www.sourcebot.dev/blog/review-agent-learnings", "title": "I built an AI code review agent in a few hours, here's what I learned", "date_modified": "2025-05-06T19:39:44.000Z" }, { "content_html": "Comments", "url": "https://www.wiz.io/blog/github-actions-security-guide", "title": "How to Harden GitHub Actions: The Unofficial Guide", "date_modified": "2025-05-06T02:07:42.000Z" }, { "content_html": "Comments", "url": "https://www.wiz.io/blog/github-actions-security-guide", "title": "How to harden GitHub Actions", "date_modified": "2025-05-06T02:07:42.000Z" }, { "content_html": "Comments", "url": "https://blog.yaakov.online/replacing-kubernetes-with-systemd/", "title": "Replacing Kubernetes with systemd (2024)", "date_modified": "2025-05-05T20:40:14.000Z" }, { "content_html": "

Has your browsing experience ever been disrupted by this error page? Sometimes Cloudflare returns \"Error 500\" when our servers cannot respond to your web request. This inability to respond could have several potential causes, including problems caused by a bug in one of the services that make up Cloudflare's software stack.

\n
\n \n

We know that our testing platform will inevitably miss some software bugs, so we built guardrails to gradually and safely release new code before a feature reaches all users. Health Mediated Deployments (HMD) is Cloudflare’s data-driven solution to automating software updates across our global network. HMD works by querying Thanos, a system for storing and scaling Prometheus metrics. Prometheus collects detailed data about the performance of our services, and Thanos makes that data accessible across our distributed network. HMD uses these metrics to determine whether new code should continue to roll out, pause for further evaluation, or be automatically reverted to prevent widespread issues.

Cloudflare engineers configure signals from their service, such as alerting rules or Service Level Objectives (SLOs). For example, the following Service Level Indicator (SLI) checks the rate of HTTP 500 errors over 10 minutes returned from a service in our software stack.

\n 
sum(rate(http_request_count{code=\"500\"}[10m])) / sum(rate(http_request_count[10m]))
\n

An SLO is a combination of an SLI and an objective threshold. For example, the service returns 500 errors <0.1% of the time.

If the success rate is unexpectedly decreasing where the new code is running, HMD reverts the change in order to stabilize the system, reacting before humans even know what Cloudflare service was broken. Below, HMD recognizes the degradation in signal in an early release stage and reverts the code back to the prior version to limit the blast radius.

\n
\n \n

\nCloudflare’s network serves millions of requests per second across diverse geographies. How do we know that HMD will react quickly the next time we accidentally release code that contains a bug? HMD performs a testing strategy called backtesting, outside the release process, which uses historical incident data to test how long it would take to react to degrading signals in a future release.

We use Thanos to join thousands of small Prometheus deployments into a single unified query layer while keeping our monitoring reliable and cost-efficient. To backfill historical incident metric data that has fallen out of Prometheus’ retention period, we use our object storage solution, R2.

Today, we store 4.5 billion distinct time series for a year of retention, which results in roughly 8 petabytes of data in 17 million objects distributed all over the globe.

\n
\n \n
\n
\n

Making it work at scale

\n \n \n \n
\n

To give a sense of scale, we can estimate the impact of a batch of backtests:

Thus, in aggregate, a batch can translate to hundreds of thousands of PromQL queries and millions of requests to R2. Initially, batch runs would take about 30 hours to complete but through blood, sweat, and tears, we were able to cut this down to 2 hours.

Let’s review how we made this processing more efficient.

\n
\n

Recording rules

\n \n \n \n
\n

HMD slices our fleet of machines across multiple dimensions. For the purposes of this post, let’s refer to them as “tier” and “color”. Given a pair of tier and color, we would use the following PromQL expression to find the machines that make up this combination:

\n 
group by (instance, datacenter, tier, color) (\n  up{job=\"node_exporter\"}\n  * on (datacenter) group_left(tier) datacenter_metadata{tier=\"tier3\"}\n  * on (instance) group_left(color) server_metadata{color=\"green\"}\n  unless on (instance) (machine_in_maintenance == 1)\n  unless on (datacenter) (datacenter_disabled == 1)\n)
\n

Most of these series have a cardinality of approximately the number of machines in our fleet. That’s a substantial amount of data we need to fetch from object storage and transmit home for query evaluation, as well as a significant number of series we need to decode and join together.

Since this is a fairly common query that is issued in every HMD run, it makes sense to precompute it. In the Prometheus ecosystem, this is commonly done with recording rules:

\n 
hmd:release_scopes:info{tier=\"tier3\", color=\"green\"}
\n

Aside from looking much cleaner, this also reduces the load at query time significantly. Since all the joins involved can only have matches within a data center, it is well-defined to evaluate those rules directly in the Prometheus instances inside the data center itself.

Compared to the original query, the cardinality we need to deal with now scales with the size of the release scope instead of the size of the entire fleet.

This is significantly cheaper and also less likely to be affected by network issues along the way, which in turn reduces the amount that we need to retry the query, on average. 

\n
\n

Distributed query processing

\n \n \n \n
\n \n
\n \n

HMD and the Thanos Querier, depicted above, are stateless components that can run anywhere, with highly available deployments in North America and Europe. Let us quickly recap what happens when we evaluate the SLI expression from HMD in our introduction:

\n 
sum(rate(http_request_count{code=\"500\"}[10m]))\n/ \nsum(rate(http_request_count[10m]))
\n

Upon receiving this query from HMD, the Thanos Querier will start requesting raw time series data for the “http_requests_total” metric from its connected Thanos Sidecar and Thanos Store instances all over the world, wait for all the data to be transferred to it, decompress it, and finally compute its result:

\n
\n \n

While this works, it is not optimal for several reasons. We have to wait for raw data from thousands of data sources all over the world to arrive in one location before we can even start to decompress it, and then we are limited by all the data being processed by one instance. If we double the number of data centers, we also need to double the amount of memory we allocate for query evaluation.

Many SLIs come in the form of simple aggregations, typically to boil down some aspect of the service's health to a number, such as the percentage of errors. As with the aforementioned recording rule, those aggregations are often distributive — we can evaluate them inside the data center and coalesce the sub-aggregations again to arrive at the same result.

To illustrate, if we had a recording rule per data center, we could rewrite our example like this:

\n 
sum(datacenter:http_request_count:rate10m{code=\"500\"})\n/ \nsum(datacenter:http_request_count:rate10m)
\n

This would solve our problems, because instead of requesting raw time series data for high-cardinality metrics, we would request pre-aggregated query results. Generally, these pre-aggregated results are an order of magnitude less data that needs to be sent over the network and processed into a final result.

However, recording rules come with a steep write-time cost in our architecture, evaluated frequently across thousands of Prometheus instances in production, just to speed up a less frequent ad-hoc batch process. Scaling recording rules alongside our growing set of service health SLIs quickly would be unsustainable. So we had to go back to the drawing board.

It would be great if we could evaluate data center-scoped queries remotely and coalesce their result back again — for arbitrary queries and at runtime. To illustrate, we would like to evaluate our example like this:

\n 
(sum(rate(http_requests_total{status=\"500\", datacenter=\"dc1\"}[10m])) + ...)\n/\n(sum(rate(http_requests_total{datacenter=\"dc1\"}[10m])) + ...)
\n

This is exactly what Thanos’ distributed query engine is capable of doing. Instead of requesting raw time series data, we request data center scoped aggregates and only need to send those back home where they get coalesced back again into the full query result:

\n
\n \n

Note that we ensure all the expensive data paths are as short as possible by utilizing R2 location hints to specify the primary access region.\n

\n
\n \n
\n
\n \n

To measure the effectiveness of this approach, we used Cloudprober and wrote probes that evaluate the relatively cheap, but still global, query count(node_uname_info).

\n 
sum(thanos_cloudprober_latency:rate6h{component=\"thanos-central\"})\n/\nsum(thanos_cloudprober_latency:rate6h{component=\"thanos-distributed\"})
\n

In the graph below, the y-axis represents the speedup of the distributed execution deployment relative to the centralized deployment. On average, distributed execution responds 3–5 times faster to probes.

\n
\n \n

Anecdotally, even slightly more complex queries quickly time out or even crash our centralized deployment, but they still can be comfortably computed by the distributed one. For a slightly more expensive query like count(up) for about 17 million scrape jobs, we had difficulty getting the centralized querier to respond and had to scope it to a single region, which took about 42 seconds:

\n
\n \n

Meanwhile, our distributed queriers were able to return the full result in about 8 seconds:

\n
\n \n
\n
\n

Congestion control

\n \n \n \n
\n

HMD batch processing leads to spiky load patterns that are hard to provision for. In a perfect world, it would issue a steady and predictable stream of queries. At the same time, HMD batch queries have lower priority to us than the queries that on-call engineers issue to triage production problems. We tackle both of those problems by introducing an adaptive priority-based concurrency control mechanism. After reading Netflix’s work on adaptive concurrency limits, we implemented a similar proxy to dynamically limit batch request flow when Thanos SLOs start to degrade. For example, one such SLO is its cloudprober failure rate over the last minute:

\n 
sum(thanos_cloudprober_fail:rate1m)\n/\n(sum(thanos_cloudprober_success:rate1m) + sum(thanos_cloudprober_fail:rate1m))
\n

We apply jitter, a random delay, to smooth query spikes inside the proxy. Since batch processing prioritizes overall query throughput over individual query latency, jitter helps HMD send a burst of queries, while allowing Thanos to process queries gradually over several minutes. This reduces instantaneous load on Thanos, improving overall throughput, even if individual query latency increases. Meanwhile, HMD encounters fewer errors, minimizing retries and boosting batch efficiency.

Our solution simulates how TCP’s congestion control algorithm, additive increase/multiplicative decrease, works. When the proxy server receives a successful request from Thanos, it allows one more concurrent request through next time. If backpressure signals breach defined thresholds, the proxy limits the congestion window proportional to the failure rate.

\n
\n \n

As the failure rate increases past the “warn” threshold, approaching the “emergency” threshold, the proxy gets exponentially closer to allowing zero additional requests through the system. However, to prevent bad signals from halting all traffic, we cap the loss with a configured minimum request rate.

\n
\n

Columnar experiments

\n \n \n \n
\n

Because Thanos deals with Prometheus TSDB blocks that were never designed for being read over a slow medium like object storage, it does a lot of random I/O. Inspired by this excellent talk, we started storing our time series data in Parquet files, with some promising preliminary results. This project is still too early to draw any robust conclusions, but we wanted to share our implementation with the Prometheus community, so we are publishing our experimental object storage gateway as parquet-tsdb-poc on GitHub.

\n
\n

Conclusion

\n \n \n \n
\n

We built Health Mediated Deployments (HMD) to enable safe and reliable software releases while pushing the limits of our observability infrastructure. Along the way, we significantly improved Thanos’ ability to handle high-load queries, reducing batch runtimes by 15x.

But this is just the beginning. We’re excited to continue working with the observability, resiliency, and R2 teams to push our infrastructure to its limits — safely and at scale. As we explore new ways to enhance observability, one exciting frontier is optimizing time series storage for object storage.

We’re sharing this work with the community as an open-source proof of concept. If you’re interested in exploring Parquet-based time series storage and its potential for large-scale observability, check out the GitHub project linked above.

", "url": "https://blog.cloudflare.com/safe-change-at-any-scale/", "title": "Scaling with safety: Cloudflare's approach to global service health metrics and software releases", "date_modified": "2025-05-05T13:20:57.162Z" }, { "content_html": "Comments", "url": "https://www.ubicloud.com/blog/building-burstables-cpu-slicing-with-cgroups", "title": "Building Burstables: CPU slicing with cgroups", "date_modified": "2025-05-02T16:45:29.000Z" }, { "content_html": "Comments", "url": "https://suno.com/explore/", "title": "Suno v4.5", "date_modified": "2025-05-02T13:18:26.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/integrations", "title": "Claude can now connect to your world", "date_modified": "2025-05-01T16:02:14.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/mscheidegger/minidisc", "title": "minidisc: Zero-config service discovery for Tailscale networks", "date_modified": "2025-05-01T14:53:45.000Z" }, { "content_html": "Comments", "url": "https://github.com/deepseek-ai/DeepSeek-Prover-V2", "title": "DeepSeek-Prover-V2", "date_modified": "2025-04-30T16:23:28.000Z" }, { "content_html": "Comments", "url": "https://github.com/neuroxhq/helm-chart-neurox-control", "title": "Show HN: Neurox – GPU Observability for AI Infra", "date_modified": "2025-04-29T18:02:03.000Z" }, { "content_html": "Comments", "url": "https://jepsen.io/analyses/amazon-rds-for-postgresql-17.4", "title": "Jepsen: Amazon RDS for PostgreSQL 17.4", "date_modified": "2025-04-29T14:30:11.000Z" }, { "content_html": "Comments", "url": "https://medium.com/gitconnected/mission-impossible-managing-ai-agents-in-the-real-world-f8e7834833af", "title": "Mission Impossible: Managing AI Agents in the Real World", "date_modified": "2025-04-29T13:54:25.000Z" }, { "content_html": "Comments", "url": "https://rnikhil.com/2025/04/25/sales-outbound-ai-dead", "title": "Is outbound going to die?", "date_modified": "2025-04-28T17:28:09.000Z" }, { "content_html": "Comments", "url": "https://github.com/mr-karan/logchef", "title": "Show HN: Logchef – Schema-agnostic log viewer for ClickHouse", "date_modified": "2025-04-27T15:15:40.000Z" }, { "content_html": "Comments", "url": "https://catalins.tech/how-i-setup-new-macbooks/", "title": "Easily setup new MacBooks", "date_modified": "2025-04-25T09:57:46.000Z" }, { "content_html": "Comments", "url": "https://motherduck.com/blog/introducing-instant-sql/", "title": "Instant SQL for results as you type in DuckDB UI", "date_modified": "2025-04-24T13:23:26.000Z" }, { "content_html": "Comments", "url": "https://iamcharliegraham.substack.com/publish/post/161906169", "title": "The Future of MCPs", "date_modified": "2025-04-23T17:12:58.000Z" }, { "content_html": "Comments", "url": "https://docs.fiveonefour.com/moose", "title": "Show HN: Moose – OSS framework to build analytical back ends with ClickHouse", "date_modified": "2025-04-23T16:41:23.000Z" }, { "content_html": "Comments", "url": "https://koomen.dev/essays/horseless-carriages/", "title": "AI Horseless Carriages", "date_modified": "2025-04-23T16:19:56.000Z" }, { "content_html": "Comments", "url": "https://www.featureform.com/post/deploy-mcp-on-aws-lambda-with-mcpengine", "title": "MCP on AWS Lambda with MCPEngine", "date_modified": "2025-04-23T16:17:04.000Z" }, { "content_html": "Comments", "url": "https://blog.atuin.sh/atuin-desktop-runbooks-that-run/", "title": "Atuin Desktop: Runbooks That Run", "date_modified": "2025-04-22T20:54:52.000Z" }, { "content_html": "Comments", "url": "https://github.com/alexykn/sapphire", "title": "Sapphire: Rust based package manager for macOS", "date_modified": "2025-04-22T18:39:20.000Z" }, { "content_html": "Comments", "url": "https://clickhouse.com/blog/clickhouse-gets-lazier-and-faster-introducing-lazy-materialization", "title": "ClickHouse gets lazier (and faster): Introducing lazy materialization", "date_modified": "2025-04-22T16:03:32.000Z" }, { "content_html": "

Software development moves fast – really fast. It can also involve multiple teams working from different locations around the world. However, while speed and collaboration can be great for developers and businesses, they can also create security challenges. 

\n\n\n\n

With more entry points and less time to catch potential threats, each commit, build, and deployment is another opportunity for something to go wrong. Whether that’s a security breach, malicious attack, or accidental exposure, the impact can ripple through your chain and burden every application.

\n\n\n\n

That’s where CI/CD security comes in. Learn what securing your CI/CD pipeline means for your team, the main risks you need to be aware of, and the practical steps to safeguard your flow.

\n\n\n\n

What is CI/CD security, and why is it important?

\n\n\n\n

CI/CD security is a set of practices and controls that protects the entire software delivery process. It prioritizes keeping your code safe from the very start, is built in rather than a separate phase, and is integral to DevSecOps.

\n\n\n\n

Your CI/CD pipeline has access to tons of sensitive information, including codebases, credentials, and production environments. If compromised, attackers could inject malicious code, steal data, or even gain access to your systems (as they did in the SolarWinds attack).

\n\n\n\n

Aside from these catastrophic breaches, proper CI/CD security helps prevent mistakes, which could expose sensitive data or introduce vulnerabilities. Malicious employee or contractor behavior shouldn’t be overlooked here, either – 20% of businesses cited this as a cause of their data breaches. CI/CD security is both a shield and a safety net in one.

\n\n\n\n
\n\n\n\n

With development automation, changes can go from a laptop to production in minutes, and CI/CD security needs to ensure it doesn’t slow down the process. Acceleration is great for business agility, but giving attackers a fast track to your systems is hazardous. In fact, less than 10% of companies in 2022 had implemented hack monitoring in their software development lifecycle. 

\n\n\n\n

However, get CI/CD security right, and you can have both speed and reliability.

\n\n\n\n

CI/CD pipeline security threats

\n\n\n\n

Your CI/CD pipeline has several potential weak points, including: 

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

These components face threats from various angles, such as: 

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

The interconnected nature of CI/CD means that compromising just one part can affect everything in the system.

\n\n\n\n
\n\n\n\n

Tips for securing your CI/CD pipeline

\n\n\n\n

The most effective CI/CD security involves building multiple layers of protection throughout your pipeline. Rather than implementing a single tool or simply following a checklist, you should set up security checkpoints at every stage.

\n\n\n\n

Employ CI/CD access controls

\n\n\n\n

Protect your pipeline by implementing strict access controls and applying the principle of least privilege. 

\n\n\n\n

Use role-based access control (RBAC) to ensure team members only have the access they absolutely need for their specific roles. To prevent unauthorized code changes, set up mandatory code reviews, enable branch protection rules, and use signed commits.

\n\n\n\n

Remember to regularly audit these permissions and remove access when team members leave.

\n\n\n\n

Manage secrets effectively

\n\n\n\n

Never, ever hardcode credentials into your pipeline configurations or code. Instead, use dedicated secrets management tools (such as HashiCorp Vault) to securely store and manage sensitive information.

\n\n\n\n

Rotate these credentials regularly (ideally automatically) and ensure secrets are encrypted both in transit and at rest. It’s also best to use temporary credentials where possible.

\n\n\n\n

Integrated security testing

\n\n\n\n

Make security testing a natural part of your pipeline by putting multiple testing layers in place.

\n\n\n\n

Certain tools can help you catch vulnerabilities before they reach production:

\n\n\n\n\n\n\n\n

Configure these tests to run automatically with each build and block deployments if any security issues are found.

\n\n\n\n
\n\n\n\n

Secure the development and deployment environment

\n\n\n\n

Ensure your build environments are as secure as your production systems – they’re just as important, if not more. 

\n\n\n\n

Harden your build servers by removing unnecessary services, keeping systems patched, and using minimal base images. Implement network segmentation to isolate build environments from each other and other systems.

\n\n\n\n

If you can, consider using temporary infrastructure. This method allows you to create fresh environments for each build and destroy them afterward.

\n\n\n\n

Automate security scans

\n\n\n\n

Set up automated security scanning throughout your pipeline. Use container scanners to check for vulnerabilities in container images, dependency checkers to identify known vulnerabilities in libraries, and registry scanners to ensure the security of stored artifacts. 

\n\n\n\n

Establish vulnerability thresholds (what level is considered suspicious or a threat) and automatically stop deployments that don’t meet your security standards. Schedule regular scans of your artifacts to ensure you’re aware of new or emerging vulnerabilities.

\n\n\n\n

Monitor and alert

\n\n\n\n

Implement comprehensive monitoring for your CI/CD pipeline. Track all activities and watch for unusual patterns like builds at odd hours, unexpected configuration changes, strange resource usage, and deployment events.

\n\n\n\n

Use detailed logging and set up alerts, making sure your team knows how to respond if something suspicious is found. Security information and event management (SIEM) are great CI/CD security tools – they correlate security events and enable real-time threat detection and response.

\n\n\n\n

Perform regular security audits and assessments

\n\n\n\n

Regularly test your CI/CD pipeline security using different methods:

\n\n\n\n\n\n\n\n

Check your compliance with your local security standards and regulations, and update your controls based on the results of your assessments.

\n\n\n\n

How TeamCity can help

\n\n\n\n

Security in your CI/CD pipeline is a must for protecting your software supply chain. While the threats are real, with the right tools and practices, you can build and deploy software securely without slowing down your team or minimizing their efforts.

\n\n\n\n

TeamCity makes this easier with security features that grow with your needs.

\n\n\n\n
TeamCity On-PremisesTeamCity Cloud
🖥️ Installed and fully managed by your team☁️ Hosted and managed by JetBrains
🔐 Full control over infrastructure and network🔒 Zero-maintenance, secure-by-default CI/CD environment
🗝️ SSH key management🗝️ SSH key management
🔄 Custom secrets management integrations (e.g., HashiCorp Vault, AWS KMS)🔄 Custom secrets management integrations (e.g., HashiCorp Vault, AWS KMS)
📦 Artifact storage and access managed internally📦 Secure artifact storage with access control
🔍 Customizable logging and monitoring tools📜 Built-in user audit logs and integrated monitoring
🔧 Highly customizable for specific compliance needs✅ Compliant with industry standards and suitable for regulated industries
👥 Ideal for teams with strict infrastructure or data residency policies🏢 Great for teams who want secure CI/CD without infrastructure management
\n\n\n\n

Deliver secure software without compromising on speed or performance. Try TeamCity for free now.

\n\n\n
\n

💡See also: [Whitepaper] 9 Ways to Prevent a Supply Chain Attack on Your CI/CD Server

\n
", "url": "https://blog.jetbrains.com/teamcity/2025/04/ci-cd-security-best-practices/", "title": "CI/CD Security Best Practices", "date_modified": "2025-04-22T12:57:38.000Z" }, { "content_html": "

Comments

", "url": "https://skiplabs.io/blog/event-hidden-arch", "title": "Event-Hidden Architectures", "date_modified": "2025-04-22T12:53:45.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/engineering/claude-code-best-practices", "title": "Claude Code Best Practices", "date_modified": "2025-04-19T10:48:30.000Z" }, { "content_html": "Comments", "url": "https://spiraldb.com/post/so-you-want-to-use-object-storage", "title": "Achieveing lower latencies with S3 object storage", "date_modified": "2025-04-19T10:19:49.000Z" }, { "content_html": "Comments", "url": "https://www.bugsink.com/blog/why-i-gave-up-on-self-hosted-sentry/", "title": "I gave up on self-hosted Sentry", "date_modified": "2025-04-18T07:24:10.000Z" }, { "content_html": "Comments", "url": "https://github.com/plandex-ai/plandex", "title": "Show HN: Plandex v2 – open source AI coding agent for large projects and tasks", "date_modified": "2025-04-16T21:26:42.000Z" }, { "content_html": "Comments", "url": "https://www.bloomberg.com/news/articles/2025-04-16/openai-said-to-be-in-talks-to-buy-windsurf-for-about-3-billion", "title": "OpenAI in Talks to Buy Windsurf for About $3B", "date_modified": "2025-04-16T18:24:25.000Z" }, { "content_html": "Comments", "url": "https://devblogs.microsoft.com/oldnewthing/20250411-00/?p=111066", "title": "The case of the UI thread that hung in a kernel call", "date_modified": "2025-04-15T17:13:31.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=43692476", "title": "Launch HN: Mrge.io (YC X25) – Cursor for code review", "date_modified": "2025-04-15T13:34:21.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=43692476", "title": "Launch HN: mrge.io (YC X25) – Cursor for code review", "date_modified": "2025-04-15T13:34:21.000Z" }, { "content_html": "Comments", "url": "https://www.infoworld.com/article/3849245/github-suffers-a-cascading-supply-chain-attack-compromising-ci-cd-secrets.html", "title": "GitHub suffers a cascading supply chain attack compromising CI/CD secrets", "date_modified": "2025-04-15T11:31:26.000Z" }, { "content_html": "Comments", "url": "https://github.com/nathanhleung/protobuf-ts-types", "title": "Show HN: Zero-codegen, no-compile TypeScript type inference from Protobufs", "date_modified": "2025-04-14T15:41:03.000Z" }, { "content_html": "Comments", "url": "https://github.com/meilisearch/meilisearch", "title": "Meilisearch – search engine API bringing AI-powered hybrid search", "date_modified": "2025-04-14T12:46:45.000Z" }, { "content_html": "Comments", "url": "https://github.com/basecamp/gh-signoff", "title": "Local CI. Sign off on your own work", "date_modified": "2025-04-14T01:12:44.000Z" }, { "content_html": "Comments", "url": "https://substack.com/home/post/p-161145826", "title": "Dev Tools Honeytrap: Why We Can't Stop Building Tools Nobody Buys", "date_modified": "2025-04-11T23:37:45.000Z" }, { "content_html": "Comments", "url": "https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/random-for-clones.md", "title": "Firecracker Entropy for VM Clones", "date_modified": "2025-04-11T22:46:25.000Z" }, { "content_html": "Comments", "url": "https://stevana.github.io/erlangs_not_about_lightweight_processes_and_message_passing.html", "title": "Erlang's not about lightweight processes and message passing", "date_modified": "2025-04-11T15:50:49.000Z" }, { "content_html": "Comments", "url": "https://stevana.github.io/erlangs_not_about_lightweight_processes_and_message_passing.html", "title": "Erlang's not about lightweight processes and message passing (2023)", "date_modified": "2025-04-11T15:50:49.000Z" }, { "content_html": "

At re:Invent 2023, we introduced Amazon S3 Express One Zone, a high-performance, single-Availability Zone (AZ) storage class purpose-built to deliver consistent single-digit millisecond data access for your most frequently accessed data and latency-sensitive applications.

\n

S3 Express One Zone delivers data access speed up to 10 times faster than S3 Standard, and it can support up to 2 million GET transactions per second (TPS) and up to 200,000 PUT TPS per directory bucket. This makes it ideal for performance-intensive workloads such as interactive data analytics, data streaming, media rendering and transcoding, high performance computing (HPC), and AI/ML trainings. Using S3 Express One Zone, customers like Fundrise, Aura, Lyrebird, Vivian Health, and Fetch improved the performance and reduced the costs of their data-intensive workloads.

\n

Since launch, we’ve introduced a number of features for our customers using S3 Express One Zone. For example, S3 Express One Zone started to support object expiration using S3 Lifecycle to expire objects based on age to help you automatically optimize storage costs. In addition, your log-processing or media-broadcasting applications can directly append new data to the end of existing objects and then immediately read the object, all within S3 Express One Zone.

\n

Today we’re announcing that, effective April 10, 2025, S3 Express One Zone has reduced storage prices by 31 percent, PUT request prices by 55 percent, and GET request prices by 85 percent. In addition, S3 Express One Zone has reduced the per-GB charges for data uploads and retrievals by 60 percent, and these charges now apply to all bytes transferred rather than just portions of requests greater than 512 KB.

\n

Here is a price reduction table in the US East (N. Virginia) Region:

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
PricePreviousNewPrice reduction
Storage
(per GB-Month)		$0.16$0.1131%
Writes
(PUT requests)		$0.0025 per 1,000 requests up to 512 KB$0.00113 per 1,000 requests55%
Reads
(GET requests)		$0.0002 per 1,000 requests up to 512 KB$0.00003 per 1,000 requests85%
Data upload
(per GB)		$0.008$0.003260%
Data retrievals
(per GB)		$0.0015$0.000660%
\n

For S3 Express One Zone pricing examples, go to the S3 billing FAQs or use the AWS Pricing Calculator.

\n

These pricing reductions apply to S3 Express One Zone in all AWS Regions where the storage class is available: US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Mumbai), Asia Pacific (Tokyo), Europe (Ireland), and Europe (Stockholm) Regions. To learn more, visit the Amazon S3 pricing page and S3 Express One Zone in the AWS Documentation.

\n

Give S3 Express One Zone a try in the S3 console today and send feedback to AWS re:Post for Amazon S3 or through your usual AWS Support contacts.

\n

— Channy

\n
\n

How is the News Blog doing? Take this 1 minute survey!

\n

(This survey is hosted by an external company. AWS handles your information as described in the AWS Privacy Notice. AWS will own the data gathered via this survey and will not share the information collected with survey respondents.)

", "url": "https://aws.amazon.com/blogs/aws/up-to-85-price-reductions-for-amazon-s3-express-one-zone/", "title": "Announcing up to 85% price reductions for Amazon S3 Express One Zone", "date_modified": "2025-04-10T21:04:19.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/r2-data-catalog-public-beta/", "title": "Cloudflare R2 Data Catalog: Managed Apache Iceberg tables with zero egress fees", "date_modified": "2025-04-10T14:30:22.000Z" }, { "content_html": "Comments", "url": "https://www.docker.com/blog/introducing-docker-model-runner/", "title": "Docker Model Runner", "date_modified": "2025-04-10T14:10:28.000Z" }, { "content_html": "Comments", "url": "https://withaqua.com", "title": "Show HN: Aqua Voice 2 – Fast Voice Input for Mac and Windows", "date_modified": "2025-04-09T16:31:06.000Z" }, { "content_html": "Comments", "url": "https://netflixtechblog.com/how-netflix-accurately-attributes-ebpf-flow-logs-afe6d644a3bc", "title": "How Netflix Accurately Attributes eBPF Flow Logs", "date_modified": "2025-04-08T18:21:49.000Z" }, { "content_html": "Comments", "url": "https://github.com/coroot/coroot", "title": "Show HN: Coroot – eBPF-based, open source observability with actionable insights", "date_modified": "2025-04-08T16:49:40.000Z" }, { "content_html": "Comments", "url": "https://yossarian.net/til/post/any-program-can-be-a-github-actions-shell/", "title": "Any program can be a GitHub Actions shell", "date_modified": "2025-04-08T01:20:10.000Z" }, { "content_html": "

Comments

", "url": "https://narang99.github.io/2025-03-22-monorepo-bazel-jenkins/", "title": "Beautiful CI for Bazel", "date_modified": "2025-04-07T22:34:22.000Z" }, { "content_html": "Comments", "url": "https://www.designboom.com/design/severance-closer-look-mid-century-brutalist-retro-futuristic-universe-lumon-03-21-2025/", "title": "Severance: A closer look into the mid-century, brutalist universe of Lumon", "date_modified": "2025-04-07T19:40:39.000Z" }, { "content_html": "

It’s not a secret that at Cloudflare we are bullish on the future of agents. We’re excited about a future where AI can not only co-pilot alongside us, but where we can actually start to delegate entire tasks to AI. 

While it hasn’t been too long since we first announced our Agents SDK to make it easier for developers to build agents, building towards an agentic future requires continuous delivery towards this goal. Today, we’re making several announcements to help accelerate agentic development, including:

\n
\n

New MCP capabilities in Agents SDK

\n \n \n \n
\n

AI agents can now connect to and interact with external services through MCP (Model Context Protocol). We’ve updated the Agents SDK to allow you to build a remote MCP client into your AI agent, with all the components — authentication flows, tool discovery, and connection management — built-in for you.

This allows you to build agents that can:

  1. Prompt the end user to grant access to a 3rd party service (MCP server).

  2. Use tools from these external services, acting on behalf of the end user.

  3. Call MCP servers from Workflows, scheduled tasks, or any part of your agent.

  4. Connect to multiple MCP servers and automatically discover new tools or capabilities presented by the 3rd party service.

\n
\n \n

MCP (Model Context Protocol) — first introduced by Anthropic — is quickly becoming the standard way for AI agents to interact with external services, with providers like OpenAI, Cursor, and Copilot adopting the protocol.

We recently announced support for building remote MCP servers on Cloudflare, and added an McpAgent class to our Agents SDK that automatically handles the remote aspects of MCP: transport and authentication/authorization. Now, we’re excited to extend the same capabilities to agents acting as MCP clients.

\n
\n \n

Want to see it in action? Use the button below to deploy a fully remote MCP client that can be used to connect to remote MCP servers.

\n

\n
\n

AI Agents can now act as remote MCP clients, with transport and auth included

\n \n \n \n
\n

AI agents need to connect to external services to access tools, data, and capabilities beyond their built-in knowledge. That means AI agents need to be able to act as remote MCP clients, so they can connect to remote MCP servers that are hosting these tools and capabilities. 

We’ve added a new class, MCPClientManager, into the Agents SDK to give you all the tooling you need to allow your AI agent to make calls to external services via MCP. The MCPClientManager class automatically handles: 

\n
\n

Granting agents access to tools with built-in auth check for MCP Clients

\n \n \n \n
\n

We've integrated the complete OAuth authentication flow directly into the Agents SDK, so your AI agents can securely connect and authenticate to any remote MCP server without you having to build authentication flow from scratch.

This allows you to give users a secure way to log in and explicitly grant access to allow the agent to act on their behalf by automatically: 

Here is an example of an agent that can securely connect to MCP servers by initializing the client manager, adding the server, and handling the authentication callbacks:

\n 
async onStart(): Promise<void> {\n  // initialize MCPClientManager which manages multiple MCP clients with optional auth\n  this.mcp = new MCPClientManager(\"my-agent\", \"1.0.0\", {\n    baseCallbackUri: `${serverHost}/agents/${agentNamespace}/${this.name}/callback`,\n    storage: this.ctx.storage,\n  });\n}\n\nasync addMcpServer(url: string): Promise<string> {\n  // Add one MCP client to our MCPClientManager\n  const { id, authUrl } = await this.mcp.connect(url);\n  // Return authUrl to redirect the user to if the user is unauthorized\n  return authUrl\n}\n\nasync onRequest(req: Request): Promise<void> {\n  // handle the auth callback after being finishing the MCP server auth flow\n  if (this.mcp.isCallbackRequest(req)) {\n    await this.mcp.handleCallbackRequest(req);\n    return new Response(\"Authorized\")\n  }\n  \n  // ...\n}
\n

Connecting to multiple MCP servers and discovering what capabilities they offer

You can use the Agents SDK to connect an MCP client to multiple MCP servers simultaneously. This is particularly useful when you want your agent to access and interact with tools and resources served by different service providers. 

The MCPClientManager class maintains connections to multiple MCP servers through the mcpConnections object, a dictionary that maps unique server names to their respective MCPClientConnection instances. 

When you register a new server connection using connect(), the manager: 

  1. Creates a new connection instance with server-specific authentication.

  2. Initializes the connections and registers for server capability notifications.

\n 
async onStart(): Promise<void> {\n  // Connect to an image generation MCP server\n  await this.mcp.connect(\"https://image-gen.example.com/mcp/sse\");\n  \n  // Connect to a code analysis MCP server\n  await this.mcp.connect(\"https://code-analysis.example.org/sse\");\n  \n  // Now we can access tools with proper namespacing\n  const allTools = this.mcp.listTools();\n  console.log(`Total tools available: ${allTools.length}`);\n}
\n

Each connection manages its own authentication context, allowing one AI agent to authenticate to multiple servers simultaneously. In addition, MCPClientManager automatically handles namespacing to prevent collisions between tools with identical names from different servers. 

For example, if both an “Image MCP Server” and “Code MCP Server” have a tool named “analyze”, they will both be independently callable without any naming conflicts.

\n
\n

Use Stytch, Auth0, and WorkOS to bring authentication & authorization to your MCP server 

\n \n \n \n
\n

With MCP, users will have a new way of interacting with your application, no longer relying on the dashboard or API as the entrypoint. Instead, the service will now be accessed by AI agents that are acting on a user’s behalf. To ensure users and agents can connect to your service securely, you’ll need to extend your existing authentication and authorization system to support these agentic interactions, implementing login flows, permissions scopes, consent forms, and access enforcement for your MCP server. 

We’re adding integrations with Stytch, Auth0, and WorkOS to make it easier for anyone building an MCP server to configure authentication & authorization for their MCP server. 

You can leverage our MCP server integration with Stytch, Auth0, and WorkOS to: 

Enforce the permissions so that agents can only invoke permitted tools. 

\n
\n \n

Get started with the examples below by using the “Deploy to Cloudflare” button to deploy the demo MCP servers in your Cloudflare account. These demos include pre-configured authentication endpoints, consent flows, and permission models that you can tailor to fit your needs. Once you deploy the demo MCP servers, you can use the Workers AI playground, a browser-based remote MCP client, to test out the end-to-end user flow. 

\n
\n

Stytch

\n \n \n \n
\n

Get started with a remote MCP server that uses Stytch to allow users to sign in with email, Google login or enterprise SSO and authorize their AI agent to view and manage their company’s OKRs on their behalf. Stytch will handle restricting the scopes granted to the AI agent based on the user’s role and permissions within their organization. When authorizing the MCP Client, each user will see a consent page that outlines the permissions that the agent is requesting that they are able to grant based on their role.

\n

For more consumer use cases, deploy a remote MCP server for a To Do app that uses Stytch for authentication and MCP client authorization. Users can sign in with email and immediately access the To Do lists associated with their account, and grant access to any AI assistant to help them manage their tasks.

\n

Regardless of use case, Stytch allows you to easily turn your application into an OAuth 2.0 identity provider and make your remote MCP server into a Relying Party so that it can easily inherit identity and permissions from your app. To learn more about how Stytch is enabling secure authentication to remote MCP servers, read their blog post.

“One of the challenges of realizing the promise of AI agents is enabling those agents to securely and reliably access data from other platforms. Stytch Connected Apps is purpose-built for these agentic use cases, making it simple to turn your app into an OAuth 2.0 identity provider to enable secure access to remote MCP servers. By combining Cloudflare Workers with Stytch Connected Apps, we're removing the barriers for developers, enabling them to rapidly transition from AI proofs-of-concept to secure, deployed implementations.” — Julianna Lamb, Co-Founder & CTO, Stytch.

\n
\n

Auth0

\n \n \n \n
\n

Get started with a remote MCP server that uses Auth0 to authenticate users through email, social logins, or enterprise SSO to interact with their todos and personal data through AI agents. The MCP server securely connects to API endpoints on behalf of users, showing exactly which resources the agent will be able to access once it gets consent from the user. In this implementation, access tokens are automatically refreshed during long running interactions.

To set it up, first deploy the protected API endpoint:

\n

Then, deploy the MCP server that handles authentication through Auth0 and securely connects AI agents to your API endpoint. 

\n

\"Cloudflare continues to empower developers building AI products with tools like AI Gateway, Vectorize, and Workers AI. The recent addition of Remote MCP servers further demonstrates that Cloudflare Workers and Durable Objects are a leading platform for deploying serverless AI. We’re very proud that Auth0 can help solve the authentication and authorization needs for these cutting-edge workloads.\" — Sandrino Di Mattia, Auth0 Sr. Director, Product Architecture.

\n
\n

WorkOS

\n \n \n \n
\n

Get started with a remote MCP server that uses WorkOS's AuthKit to authenticate users and manage the permissions granted to AI agents. In this example, the MCP server dynamically exposes tools based on the user's role and access rights. All authenticated users get access to the add tool, but only users who have been assigned the image_generation permission in WorkOS can grant the AI agent access to the image generation tool. This showcases how MCP servers can conditionally expose capabilities to AI agents based on the authenticated user's role and permission.

\n

“MCP is becoming the standard for AI agent integration, but authentication and authorization are still major gaps for enterprise adoption. WorkOS Connect enables any application to become an OAuth 2.0 authorization server, allowing agents and MCP clients to securely obtain tokens for fine-grained permission authorization and resource access. With Cloudflare Workers, developers can rapidly deploy remote MCP servers with built-in OAuth and enterprise-grade access control. Together, WorkOS and Cloudflare make it easy to ship secure, enterprise-ready agent infrastructure.” — Michael Grinich, CEO of WorkOS.

\n
\n

Hibernate-able WebSockets: put AI agents to sleep when they’re not in use

\n \n \n \n
\n

Starting today, a new improvement is landing in the McpAgent class: support for the WebSockets Hibernation API that allows your MCP server to go to sleep when it’s not receiving requests and instantly wake up when it’s needed. That means that you now only pay for compute when your agent is actually working.

We recently introduced the McpAgent class, which allows developers to build remote MCP servers on Cloudflare by using Durable Objects to maintain stateful connections for every client session. We decided to build McpAgent to be stateful from the start, allowing developers to build servers that can remember context, user preferences, and conversation history. But maintaining client connections means that the session can remain active for a long time, even when it’s not being used. 

\n
\n

MCP Agents are hibernate-able by default

\n \n \n \n
\n

You don’t need to change your code to take advantage of hibernation. With our latest SDK update, all McpAgent instances automatically include hibernation support, allowing your stateful MCP servers to sleep during inactive periods and wake up with their state preserved when needed. 

\n
\n

How it works

\n \n \n \n
\n

When a request comes in on the Server-Sent Events endpoint, /sse, the Worker initializes a WebSocket connection to the appropriate Durable Object for the session and returns an SSE stream back to the client. All responses flow over this stream.

The implementation leverages the WebSocket Hibernation API within Durable Objects. When periods of inactivity occur, the Durable Object can be evicted from memory while keeping the WebSocket connection open. If the WebSocket later receives a message, the runtime recreates the Durable Object and delivers the message to the appropriate handler.

\n
\n

Durable Objects on free tier

\n \n \n \n
\n

To help you build AI agents on Cloudflare, we’re making Durable Objects available on the free tier, so you can start with zero commitment. With Agents SDK, your AI agents deploy to Cloudflare running on Durable Objects.

Durable Objects offer compute alongside durable storage, that when combined with Workers, unlock stateful, serverless applications. Each Durable Object is a stateful coordinator for handling client real-time interactions, making requests to external services like LLMs, and creating agentic “memory” through state persistence in zero-latency SQLite storage — all tasks required in an AI agent. Durable Objects scale out to millions of agents effortlessly, with each agent created near the user interacting with their agent for fast performance, all managed by Cloudflare. 

Zero-latency SQLite storage in Durable Objects was introduced in public beta September 2024 for Birthday Week. Since then, we’ve focused on missing features and robustness compared to pre-existing key-value storage in Durable Objects. We are excited to make SQLite storage generally available, with a 10 GB SQLite database per Durable Object, and recommend SQLite storage for all new Durable Object classes. Durable Objects free tier can only access SQLite storage.

Cloudflare’s free tier allows you to build real-world applications. On the free plan, every Worker request can call a Durable Object. For usage-based pricing, Durable Objects incur compute and storage usage with the following free tier limits.

\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
 \n

Workers Free

\n 		\n

Workers Paid

\n
\n

Compute: Requests

\n 		\n

100,000 / day

\n 		\n

1 million / month included

\n

+ $0.15 / million

\n
\n

Compute: Duration

\n 		\n

13,000 GB-s / day

\n 		\n

400,000 GB-s / month  included 

\n

+ $12.50 / million GB-s

\n
\n

Storage: Rows read

\n 		\n

5 million / day

\n 		\n

25 billion / month included

\n

+ $0.001 / million 

\n
\n

Storage: Rows written

\n 		\n

100,000 / day

\n 		\n

50 million / month included

\n

+ $1.00 / million

\n
\n

Storage: SQL stored data

\n 		\n

5 GB (total)

\n 		\n

5 GB-month included

\n

+ $0.20 / GB-month

\n
\n
\n
\n
\n

Find us at agents.cloudflare.com

\n \n \n \n
\n

We realize this is a lot of information to take in, but don’t worry. Whether you’re new to agents as a whole, or looking to learn more about how Cloudflare can help you build agents, today we launched a new site to help get you started — agents.cloudflare.com. 

Let us know what you build!

", "url": "https://blog.cloudflare.com/building-ai-agents-with-mcp-authn-authz-and-durable-objects/", "title": "Piecing together the Agent puzzle: MCP, authentication & authorization, and Durable Objects free tier", "date_modified": "2025-04-07T13:12:21.944Z" }, { "content_html": "Comments", "url": "https://tigerbeetle.com/blog/2025-02-27-why-we-designed-tigerbeetles-docs-from-scratch/", "title": "We Designed TigerBeetle's Docs from Scratch", "date_modified": "2025-04-07T12:35:03.000Z" }, { "content_html": "Comments", "url": "https://blogsystem5.substack.com/p/bazel-next-generation", "title": "The next generation of Bazel builds", "date_modified": "2025-04-06T13:40:56.000Z" }, { "content_html": "Comments", "url": "https://thewisenerd.com/blog/ext4-readdir/", "title": "The order of files in your ext4 filesystem does not matter", "date_modified": "2025-04-06T12:39:40.000Z" }, { "content_html": "

Comments

", "url": "https://huijzer.xyz/posts/jas/", "title": "Done with GitHub Actions Supply Chain Attacks", "date_modified": "2025-04-06T07:07:44.000Z" }, { "content_html": "

Comments

", "url": "https://gergely.imreh.net/blog/2025/04/the-curious-case-of-binfmt-for-x86-emulation-for-arm-docker/", "title": "The curious case of binfmt for x86 emulation for ARM Docker", "date_modified": "2025-04-04T06:28:53.000Z" }, { "content_html": "Comments", "url": "https://github.com/awslabs/mcp", "title": "AWS MCP Servers", "date_modified": "2025-04-03T11:24:10.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/pdtpartners/nix-ninja", "title": "Introducing Nix Ninja – open-source Ninja-compatible build system for Nix", "date_modified": "2025-04-03T11:10:32.000Z" }, { "content_html": "Comments", "url": "https://github.com/pdtpartners/nix-ninja", "title": "Show HN: Nix Ninja – open-source Ninja-compatible build system for Nix", "date_modified": "2025-04-03T10:47:11.000Z" }, { "content_html": "Comments", "url": "https://docs.mermaidchart.com/blog/posts/mermaid-chart-vs-code-plugin-create-and-edit-mermaid-js-diagrams-in-visual-studio-code", "title": "Show HN: Mermaid Chart VS Code Plugin: Mermaid.js Diagrams in Visual Studio Code", "date_modified": "2025-04-02T16:33:53.000Z" }, { "content_html": "Comments", "url": "https://pears.com/news/introducing-bare-actually-run-javascript-everywhere/", "title": "Bare: Run JavaScript Everywhere", "date_modified": "2025-04-02T16:18:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/aftertheflood/sparks", "title": "Sparks – A typeface for creating sparklines in text without code", "date_modified": "2025-04-02T06:44:17.000Z" }, { "content_html": "

Apple spent years ignoring RCS, allowing iPhones to offer a degraded messaging experience with Android users. This made Android folks unwelcome in many a group chat, but Apple finally started rectifying this issue last year with the addition of RCS support in iOS. It has been a slow rollout, though, with Google's mobile service only now getting support.

\n

While Apple supports RCS messaging on iPhones now, it has not exactly been enthusiastic about it. Anyone using Google Fi on an iPhone was left in the lurch even after Apple changed course. The first RCS update rolled out in iOS 18 last fall, but it only supported postpaid plans on the big three carriers. Most other wireless subscribers had to wait, including those on Google Fi, as confirmed to Ars last year. It was a suitably amusing outcome, considering Google is largely responsible for reviving the RCS standard and runs the Jibe back-end servers through which many iPhone RCS messages flow.

\n

Slowly but surely, Apple is making good on its promises to enable RCS as it gets the necessary data from carriers. The company released iOS 18.4 this week, and hiding amid the control center tweaks and priority notifications is support for RCS on Google Fi and other T-Mobile MVNOs. Some users spotted this feature in the recent beta releases, but the servers that handle RCS for Google's mobile service were not yet connectable. With the final release, Google has confirmed that RCS is ready at last.

Read full article

\n

Comments

", "url": "https://arstechnica.com/gadgets/2025/04/google-fi-users-on-iphone-finally-get-rcs-messaging/", "title": "Apple enables RCS messaging for Google Fi subscribers at last", "date_modified": "2025-04-01T20:22:53.000Z" }, { "content_html": "Comments", "url": "https://queue.acm.org/detail.cfm?id=3712057", "title": "Systems Correctness Practices at AWS: Leveraging Formal and Semi-Formal Methods", "date_modified": "2025-04-01T14:59:42.000Z" }, { "content_html": "Comments", "url": "https://www.aboutamazon.com/news/innovation-at-amazon/amazon-nova-website-sdk", "title": "Amazon introduces Nova Chat, entering the arena with ChatGPT, Claude, Grok", "date_modified": "2025-03-31T14:36:25.000Z" }, { "content_html": "Comments", "url": "https://www.dolthub.com/blog/2025-03-29-vibin/", "title": "Vibe Coding with Cursor", "date_modified": "2025-03-29T22:52:55.000Z" }, { "content_html": "Comments", "url": "https://pagedout.institute/?page=blog.php#entry-2025-03-29", "title": "Paged Out #6 is out", "date_modified": "2025-03-29T18:12:03.000Z" }, { "content_html": "Comments", "url": "https://depot.dev/blog/uncovering-disk-io-bottlenecks-github-actions-ci", "title": "Disk I/O bottlenecks in GitHub Actions", "date_modified": "2025-03-28T15:22:36.000Z" }, { "content_html": "Comments", "url": "https://world.hey.com/dhh/it-s-five-grand-a-day-to-miss-our-s3-exit-b8293563", "title": "It's five grand a day to miss our S3 exit", "date_modified": "2025-03-27T02:02:40.000Z" }, { "content_html": "Comments", "url": "https://edera.dev/stories/styrolite", "title": "Building a Linux Container Runtime from Scratch", "date_modified": "2025-03-26T20:35:46.000Z" }, { "content_html": "Comments", "url": "https://iximiuz.com/en/posts/iximiuz-labs-story/", "title": "Building a Firecracker-Powered Course Platform to Learn Docker and Kubernetes", "date_modified": "2025-03-26T20:08:32.000Z" }, { "content_html": "Comments", "url": "https://github.com/microsoft/playwright-mcp", "title": "Playwright Tools for MCP", "date_modified": "2025-03-26T19:07:39.000Z" }, { "content_html": "Comments", "url": "https://alexwlchan.net/2025/github-actions-audit/", "title": "Whose code am I running in GitHub Actions?", "date_modified": "2025-03-25T17:17:05.000Z" }, { "content_html": "Comments", "url": "https://www.npmjs.com/package/apidog-mcp-server", "title": "Show HN: We made an MCP Server so that Cursor can build anything from API Docs", "date_modified": "2025-03-24T10:21:02.000Z" }, { "content_html": "Comments", "url": "https://github.com/tsl0922/ttyd", "title": "Ttyd – Share your terminal over the web", "date_modified": "2025-03-23T17:26:11.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=43439939", "title": "Ask HN: What is the simplest data orchestration tool you've worked with?", "date_modified": "2025-03-21T19:24:54.000Z" }, { "content_html": "Comments", "url": "https://www.bigscreenvr.com/", "title": "Bigscreen Beyond 2", "date_modified": "2025-03-21T17:03:27.000Z" }, { "content_html": "

Comments

", "url": "https://garnix.io/blog/what-comes-after-gha", "title": "What Comes After GitHub Actions?", "date_modified": "2025-03-21T16:26:34.000Z" }, { "content_html": "

Connections made over cleartext HTTP ports risk exposing sensitive information because the data is transmitted unencrypted and can be intercepted by network intermediaries, such as ISPs, Wi-Fi hotspot providers, or malicious actors on the same network. It’s common for servers to either redirect or return a 403 (Forbidden) response to close the HTTP connection and enforce the use of HTTPS by clients. However, by the time this occurs, it may be too late, because sensitive information, such as an API token, may have already been transmitted in cleartext in the initial client request. This data is exposed before the server has a chance to redirect the client or reject the connection.

A better approach is to refuse the underlying cleartext connection by closing the network ports used for plaintext HTTP, and that’s exactly what we’re going to do for our customers.

Today we’re announcing that we’re closing all of the HTTP ports on api.cloudflare.com. We’re also making changes so that api.cloudflare.com can change IP addresses dynamically, in line with on-going efforts to decouple names from IP addresses, and reliably managing addresses in our authoritative DNS. This will enhance the agility and flexibility of our API endpoint management. Customers relying on static IP addresses for our API endpoints will be notified in advance to prevent any potential availability issues.

In addition to taking this first step to secure Cloudflare API traffic, we’ll release the ability for customers to opt-in to safely disabling all HTTP port traffic for their websites on Cloudflare. We expect to make this free security feature available in the last quarter of 2025.

We have consistently advocated for strong encryption standards to safeguard users’ data and privacy online. As part of our ongoing commitment to enhancing Internet security, this blog post details our efforts to enforce HTTPS-only connections across our global network. 

\n
\n

Understanding the problem

\n \n \n \n
\n

We already provide an “Always Use HTTPS” setting that can be used to redirect all visitor traffic on our customers’ domains (and subdomains) from HTTP (plaintext) to HTTPS (encrypted). For instance, when a user clicks on an HTTP version of the URL on the site (http://www.example.com), we issue an HTTP 3XX redirection status code to immediately redirect the request to the corresponding HTTPS version (https://www.example.com) of the page. While this works well for most scenarios, there’s a subtle but important risk factor: What happens if the initial plaintext HTTP request (before the redirection) contains sensitive user information?

\n
\n \n

Initial plaintext HTTP request is exposed to the network before the server can redirect to the secure HTTPS connection.

Third parties or intermediaries on shared networks could intercept sensitive data from the first plaintext HTTP request, or even carry out a Monster-in-the-Middle (MITM) attack by impersonating the web server.

One may ask if HTTP Strict Transport Security (HSTS) would partially alleviate this concern by ensuring that, after the first request, visitors can only access the website over HTTPS without needing a redirect. While this does reduce the window of opportunity for an adversary, the first request still remains exposed. Additionally, HSTS is not applicable by default for most non-user-facing use cases, such as API traffic from stateless clients. Many API clients don’t retain browser-like state or remember HSTS headers they've encountered. It is quite common practice for API calls to be redirected from HTTP to HTTPS, and hence have their initial request exposed to the network.

Therefore, in line with our culture of dogfooding, we evaluated the accessibility of the Cloudflare API (api.cloudflare.com) over HTTP ports (80, and others). In that regard, imagine a client making an initial request to our API endpoint that includes their secret API key. While we outright reject all plaintext connections with a 403 Forbidden response instead of redirecting for API traffic — clearly indicating that “Cloudflare API is only accessible over TLS” — this rejection still happens at the application layer. By that point, the API key may have already been exposed over the network before we can even reject the request. We do have a notification mechanism in place to alert customers and rotate their API keys accordingly, but a stronger approach would be to eliminate the exposure entirely. We have an opportunity to improve!

\n
\n

A better approach to API security

\n \n \n \n
\n

Any API key or token exposed in plaintext on the public Internet should be considered compromised. We can either address exposure after it occurs or prevent it entirely. The reactive approach involves continuously tracking and revoking compromised credentials, requiring active management to rotate each one. For example, when a plaintext HTTP request is made to our API endpoints, we detect exposed tokens by scanning for 'Authorization' header values.

In contrast, a preventive approach is stronger and more effective, stopping exposure before it happens. Instead of relying on the API service application to react after receiving potentially sensitive cleartext data, we can preemptively refuse the underlying connection at the transport layer, before any HTTP or application-layer data is exchanged. The preventative approach can be achieved by closing all plaintext HTTP ports for API traffic on our global network. The added benefit is that this is operationally much simpler: by eliminating cleartext traffic, there's no need for key rotation.

\n
\n \n

The transport layer carries the application layer data on top.

To explain why this works: an application-layer request requires an underlying transport connection, like TCP or QUIC, to be established first. The combination of a port number and an IP address serves as a transport layer identifier for creating the underlying transport channel. Ports direct network traffic to the correct application-layer process — for example, port 80 is designated for plaintext HTTP, while port 443 is used for encrypted HTTPS. By disabling the HTTP cleartext server-side port, we prevent that transport channel from being established during the initial \"handshake\" phase of the connection — before any application data, such as a secret API key, leaves the client’s machine.

\n
\n \n

Both TCP and QUIC transport layer handshakes are a pre-requisite for HTTPS application data exchange on the web.

Therefore, closing the HTTP interface entirely for API traffic gives a strong and visible fast-failure signal to developers that might be mistakenly accessing http://… instead of https://… with their secret API keys in the first request — a simple one-letter omission, but one with serious implications.

In theory, this is a simple change, but at Cloudflare’s global scale, implementing it required careful planning and execution. We’d like to share the steps we took to make this transition.

\n
\n

Understanding the scope

\n \n \n \n
\n

In an ideal scenario, we could simply close all cleartext HTTP ports on our network. However, two key challenges prevent this. First, as shown in the Cloudflare Radar figure below, about 2-3% of requests from “likely human” clients to our global network are over plaintext HTTP. While modern browsers prominently warn users about insecure HTTP connections and offer features to silently upgrade to HTTPS, this protection doesn't extend to the broader ecosystem of connected devices. IoT devices with limited processing power, automated API clients, or legacy software stacks often lack such safeguards entirely. In fact, when filtering on plaintext HTTP traffic that is “likely automated”, the share rises to over 16%! We continue to see a wide variety of legacy clients accessing resources over plaintext connections. This trend is not confined to specific networks, but is observable globally.

Closing HTTP ports, like port 80, across our entire IP address space would block such clients entirely, causing a major disruption in services. While we plan to cautiously start by implementing the change on Cloudflare's API IP addresses, it’s not enough. Therefore, our goal is to ensure all of our customers’ API traffic benefits from this change as well.

\n
\n \n

Breakdown of HTTP and HTTPS for ‘human’ connections

The second challenge relates to limitations posed by the longstanding BSD Sockets API at the server-side, which we have addressed using Tubular, a tool that inspects every connection terminated by a server and decides which application should receive it. Operators historically have faced a challenging dilemma: either listen to the same ports across many IP addresses using a single socket (scalable but inflexible), or maintain individual sockets for each IP address (flexible but unscalable). Luckily, Tubular has allowed us to resolve this using 'bindings', which decouples sockets from specific IP:port pairs. This creates efficient pathways for managing endpoints throughout our systems at scale, enabling us to handle both HTTP and HTTPS traffic intelligently without the traditional limitations of socket architecture.

Step 0, then, is about provisioning both IPv4 and IPv6 address space on our network that by default has all HTTP ports closed. Tubular enables us to configure and manage these IP addresses differently than others for our endpoints. Additionally, Addressing Agility and Topaz enable us to assign these addresses dynamically, and safely, for opted-in domains.

\n
\n

Moving from strategy to execution

\n \n \n \n
\n

In the past, our legacy stack would have made this transition challenging, but today’s Cloudflare possesses the appropriate tools to deliver a scalable solution, rather than addressing it on a domain-by-domain basis.

Using Tubular, we were able to bind our new set of anycast IP prefixes to our TLS-terminating proxies across the globe. To ensure that no plaintext HTTP traffic is served on these IP addresses, we extended our global iptables firewall configuration to reject any inbound packets on HTTP ports.

\n 
iptables -A INPUT -p tcp -d <IP_ADDRESS_BLOCK> --dport <HTTP_PORT> -j REJECT \n--reject-with tcp-reset\n\niptables -A INPUT -p udp -d <IP_ADDRESS_BLOCK> --dport <HTTP_PORT> -j REJECT \n--reject-with icmp-port-unreachable
\n

As a result, any connections to these IP addresses on HTTP ports are filtered and rejected at the transport layer, eliminating the need for state management at the application layer by our web proxies.

The next logical step is to update the DNS assignments so that API traffic is routed over the correct IP addresses. In our case, we encoded a new DNS policy for API traffic for the HTTPS-only interface as a declarative Topaz program in our authoritative DNS server:

\n 
- name: https_only\n exclusive: true \n config: |\n    (config\n      ([traffic_class \"API\"]\n       [ipv4 (ipv4_address “192.0.2.1”)] # Example IPv4 address\n       [ipv6 (ipv6_address “2001:DB8::1:1”)] # Example IPv6 address\n       [t (ttl 300]))\n  match: |\n    (= query_domain_class traffic_class)\n  response: |\n    (response (list ipv4) (list ipv6) t)
\n

The above policy encodes that for any DNS query targeting the ‘API traffic’ class, we return the respective HTTPS-only interface IP addresses. Topaz’s safety guarantees ensure exclusivity, preventing other DNS policies from inadvertently matching the same queries and misrouting plaintext HTTP expected domains to HTTPS-only IPs\n\napi.cloudflare.com is the first domain to be added to our HTTPS-only API traffic class, with other applicable endpoints to follow.

\n
\n

Opting-in your API endpoints

\n \n \n \n
\n

As we said above, we've started with api.cloudflare.com and our internal API endpoints to thoroughly monitor any side effects on our own systems before extending this feature to customer domains. We have deployed these changes gradually across all data centers, leveraging Topaz’s flexibility to target subsets of traffic, minimizing disruptions, and ensuring a smooth transition.

To monitor unencrypted connections for your domains, before blocking access using the feature, you can review the relevant analytics on the Cloudflare dashboard. Log in, select your account and domain, and navigate to the \"Analytics & Logs\" section. There, under the \"Traffic Served Over SSL\" subsection, you will find a breakdown of encrypted and unencrypted traffic for your site. That data can help provide a baseline for assessing the volume of plaintext HTTP connections for your site that will be blocked when you opt in. After opting in, you would expect no traffic for your site will be served over plaintext HTTP, and therefore that number should go down to zero.

\n
\n \n

Snapshot of ‘Traffic Served Over SSL’ section on Cloudflare dashboard

Towards the last quarter of 2025, we will provide customers the ability to opt in their domains using the dashboard or API (similar to enabling the Always Use HTTPS feature). Stay tuned!

\n
\n

Wrapping up

\n \n \n \n
\n

Starting today, any unencrypted connection to api.cloudflare.com will be completely rejected. Developers should not expect a 403 Forbidden response any longer for HTTP connections, as we will prevent the underlying connection to be established by closing the HTTP interface entirely. Only secure HTTPS connections will be allowed to be established.

We are also making updates to transition api.cloudflare.com away from its static IP addresses in the future. As part of that change, we will be discontinuing support for non-SNI legacy clients for Cloudflare API specifically — currently, an average of just 0.55% of TLS connections to the Cloudflare API do not include an SNI value. These non-SNI connections are initiated by a small number of accounts. We are committed to coordinating this transition and will work closely with the affected customers before implementing the change. This initiative aligns with our goal of enhancing the agility and reliability of our API endpoints.

Beyond the Cloudflare API use case, we're also exploring other areas where it's safe to close plaintext traffic ports. While the long tail of unencrypted traffic may persist for a while, it shouldn’t be forced on every site.\n\nIn the meantime, a small step like this can allow us to have a big impact in helping make a better Internet, and we are working hard to reliably bring this feature to your domains. We believe security should be free for all!

", "url": "https://blog.cloudflare.com/https-only-for-cloudflare-apis-shutting-the-door-on-cleartext-traffic/", "title": "HTTPS-only for Cloudflare APIs: shutting the door on cleartext traffic", "date_modified": "2025-03-20T13:00:00.000Z" }, { "content_html": "Comments", "url": "https://www.feldera.com/blog/the-pain-that-is-github-actions", "title": "The Pain That Is GitHub Actions", "date_modified": "2025-03-20T03:37:31.000Z" }, { "content_html": "Comments", "url": "https://duckdb.org/2025/03/14/preview-amazon-s3-tables.html", "title": "Preview: Amazon S3 Tables and Lakehouse in DuckDB", "date_modified": "2025-03-18T16:36:20.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/the-real-fail-rate-of-ebs", "title": "The Failure Rate of EBS", "date_modified": "2025-03-18T14:24:04.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/the-real-fail-rate-of-ebs", "title": "The real failure rate of EBS", "date_modified": "2025-03-18T14:24:04.000Z" }, { "content_html": "Comments", "url": "https://www.ycombinator.com/companies/depot/jobs/307RqGp-founding-developer-marketer", "title": "Depot (YC W23) is hiring a founding developer marketer (EU/US remote)", "date_modified": "2025-03-18T07:00:20.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/parkerconrad/status/1901615179718406276", "title": "Rippling sues Deel over spying", "date_modified": "2025-03-17T13:03:52.000Z" }, { "content_html": "Comments", "url": "https://github.com/macos-fuse-t/scorpi", "title": "Scorpi – A Modern Hypervisor (For macOS)", "date_modified": "2025-03-16T13:01:31.000Z" }, { "content_html": "

Comments

", "url": "https://www.fast.ai/posts/2023-05-03-mojo-launch.html", "title": "Mojo may be the biggest programming language advance in decades (2023)", "date_modified": "2025-03-16T01:14:51.000Z" }, { "content_html": "Comments", "url": "https://info.varnish-software.com/blog/tinykvm-the-fastest-sandbox", "title": "TinyKVM: Fast sandbox that runs on top of Varnish", "date_modified": "2025-03-14T02:12:11.000Z" }, { "content_html": "Comments", "url": "https://github.com/ZachGoldberg/Startup-CTO-Handbook/blob/main/StartupCTOHandbook.md", "title": "The Startup CTO's Handbook", "date_modified": "2025-03-11T22:18:42.000Z" }, { "content_html": "Comments", "url": "https://devblogs.microsoft.com/typescript/typescript-native-port/", "title": "A 10x Faster TypeScript", "date_modified": "2025-03-11T14:32:23.000Z" }, { "content_html": "Comments", "url": "https://github.com/johnbean393/Sidekick", "title": "Sidekick: Local-first native macOS LLM app", "date_modified": "2025-03-09T08:08:02.000Z" }, { "content_html": "Comments", "url": "https://github.com/PostHog/posthog/blob/master/.cursorrules", "title": "Posthog/.cursorrules", "date_modified": "2025-03-09T03:30:52.000Z" }, { "content_html": "Comments", "url": "https://www.telegraph.co.uk/us/politics/2025/03/08/us-to-cease-all-future-military-exercises-in-europe-reports/", "title": "US 'to cease all future military exercises in Europe'", "date_modified": "2025-03-09T01:45:07.000Z" }, { "content_html": "Comments", "url": "https://www.msn.com/en-us/money/companies/military-tech-startups-vie-for-billions-as-hegseth-shakes-up-pentagon-spending/ar-AA1zYgzZ", "title": "SpaceX teams up with Thiel's Palantir, Anduril on American Golden Dome", "date_modified": "2025-03-08T23:06:44.000Z" }, { "content_html": "Comments", "url": "https://havenweb.org/2022/11/02/facebook-lie.html", "title": "The Lie That Facebook Sold You", "date_modified": "2025-03-08T22:44:08.000Z" }, { "content_html": "Comments", "url": "https://github.com/bypirob/airo", "title": "Deploy from local to production (self-hosted)", "date_modified": "2025-03-08T18:54:17.000Z" }, { "content_html": "Comments", "url": "https://pola.rs/posts/polars-cloud-what-we-are-building/", "title": "Polars Cloud: The Distributed Cloud Architecture to Run Polars Anywhere", "date_modified": "2025-03-07T20:57:46.000Z" }, { "content_html": "Comments", "url": "https://rxdb.info/articles/local-first-future.html", "title": "Why Local-First Software Is the Future and Its Limitations", "date_modified": "2025-03-07T13:12:30.000Z" }, { "content_html": "Comments", "url": "https://github.com/warewulf/warewulf", "title": "Warewulf is a stateless and diskless container OS provisioning system", "date_modified": "2025-03-06T18:45:47.000Z" }, { "content_html": "Comments", "url": "https://simonwillison.net/2025/Mar/6/aider-using-uv-as-an-installer/", "title": "Aider: Using Uv as an Installer", "date_modified": "2025-03-06T03:18:06.000Z" }, { "content_html": "

One thing Shadowveil: Legend of the Five Rings does well is invoke terror. Not just the terror of an overwhelming mass of dark energy encroaching on your fortress, which is what the story suggests; more so, the terror of hoping your little computer-controlled fighters will do the smart thing and then being forced to watch, helpless, as they are consumed by algorithmic choices, bad luck, your strategies, or some combination of all three.

\n

Shadowveil, the first video game based on the more than 30-year-old Legend of the Five Rings fantasy franchise, is a roguelite auto-battler. You pick your Crab Clan hero (berserker hammer-wielder or tactical support type), train up some soldiers, and assign all of them abilities, items, and buffs you earn as you go. When battle starts, you choose which hex to start your fighters on, double-check your load-outs, then click to start and watch what happens. You win and march on, or you lose and regroup at base camp, buying some upgrades with your last run's goods.

\n
\n
\n
\n Shadowveil: Legend of the Five Rings launch trailer.\n\n
\n
\n
\n

In my impressions after roughly seven hours of playing, Shadowveil could do more to soften its learning curve, but it presents a mostly satisfying mix of overwhelming odds and achievement. What's irksome now could get patched, and what's already there is intriguing, especially for the price.

Read full article

\n

Comments

", "url": "https://arstechnica.com/gaming/2025/03/shadowveil-is-a-stylish-tough-single-player-auto-battler/", "title": "Shadowveil is a stylish, tough single-player auto-battler", "date_modified": "2025-03-05T12:00:24.000Z" }, { "content_html": "
\n
\n
\n Millie Bobby Brown and Chris Pratt star in the Netflix original film The Electric State.\n\n
\n
\n
\n

Anthony and Joe Russo have their hands full these days with the Marvel films Avengers: Doomsday and Avengers: Secret War, slated for 2026 and 2027 releases, respectively. But we'll get a chance to see another, smaller film from the directors this month on Netflix: The Electric State, adapted from the graphic novel by Swedish artist/designer Simon Stålenhag.

\n

Stålenhag's stunningly surreal neofuturistic art—featured in his narrative art books, 2014's Tales from the Loop and 2016's Things From the Flood—inspired the 2020 eight-episode series Tales From the Loop, in which residents of a rural town find themselves grappling with strange occurrences thanks to the presence of an underground particle accelerator. That adaptation captured the mood and tone of the art that inspired it and received Emmy nominations for cinematography and special visual effects.

\n

The Electric State was Stålenhag's third such book, published in 2018 and set in a similar dystopian, ravaged landscape. Paragraphs of text, accompanied by larger artworks, tell the story of a teen girl named Michelle who must travel across the country with her robot companion to find her long-lost brother, while being pursued by a federal agent. The Russo brothers acquired the rights early on and initially intended to make the film with Universal, but when the studio decided it would not be giving the film a theatrical release, Netflix bought the distribution rights.

Read full article

\n

Comments

", "url": "https://arstechnica.com/culture/2025/03/humans-and-bots-take-on-the-system-in-the-electric-state-trailer/", "title": "Netflix drops trailer for the Russo brothers’ The Electric State", "date_modified": "2025-03-03T19:21:43.000Z" }, { "content_html": "Comments", "url": "https://xeiaso.net/blog/2025/yoke-k8s/", "title": "Yoke: Infrastructure as code, but actually", "date_modified": "2025-03-02T13:56:01.000Z" }, { "content_html": "

Comments

", "url": "https://www.shadaj.me/writing/distributed-programming-stalled", "title": "Distributed Systems Programming Has Stalled", "date_modified": "2025-02-28T04:57:17.000Z" }, { "content_html": "Comments", "url": "https://app.laravel.cloud/", "title": "Laravel Cloud", "date_modified": "2025-02-24T15:26:54.000Z" }, { "content_html": "

Comments

", "url": "https://youtu.be/SKXEppEZp9M", "title": "Scrap Your ORM—Replace Your ORM With Relational Algebra", "date_modified": "2025-02-22T18:40:40.000Z" }, { "content_html": "Comments", "url": "https://blog.trailofbits.com/2025/02/21/the-1.5b-bybit-hack-the-era-of-operational-security-failures-has-arrived/", "title": "The $1.5B Bybit Hack: The Era of Operational Security Failures Has Arrived", "date_modified": "2025-02-22T17:05:27.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=43129301", "title": "Launch HN: Massdriver (YC W22) – Self-serve cloud infra without the red tape", "date_modified": "2025-02-21T16:19:12.000Z" }, { "content_html": "Comments", "url": "https://benjamintoll.com/2022/02/04/on-running-systemd-nspawn-containers/", "title": "Running Systemd-Nspawn Containers", "date_modified": "2025-02-21T08:00:54.000Z" }, { "content_html": "Comments", "url": "https://docs.docker.com/docker-hub/usage/", "title": "Docker limits unauthenticated pulls to 10/HR/IP from Docker Hub, from March 1", "date_modified": "2025-02-21T07:42:45.000Z" }, { "content_html": "

Comments

", "url": "https://docs.docker.com/docker-hub/usage/", "title": "Docker limits unauthenticated pulls to 10/hr/ip from Docker Hub, from March 1", "date_modified": "2025-02-21T07:40:41.000Z" }, { "content_html": "Comments", "url": "https://obsidian.md/blog/free-for-work/", "title": "Obsidian is now free for work", "date_modified": "2025-02-20T16:50:18.000Z" }, { "content_html": "Comments", "url": "https://yaak.app/blog/2025.1.1", "title": "Yaak 2.0 – Git, WebSockets, OAuth, and More", "date_modified": "2025-02-19T20:52:53.000Z" }, { "content_html": "Comments", "url": "https://github.com/mastra-ai/mastra", "title": "Show HN: Mastra – Open-source TypeScript agent framework", "date_modified": "2025-02-19T15:25:08.000Z" }, { "content_html": "Comments", "url": "https://github.com/mastra-ai/mastra", "title": "Show HN: Mastra – Open-source JS agent framework, by the creators of Gatsby", "date_modified": "2025-02-19T15:25:08.000Z" }, { "content_html": "Comments", "url": "https://www.ubicloud.com/blog/debugging-hetzner-uncovering-failures-with-powerstat-sensors-and-dmidecode", "title": "Debugging Hetzner: Uncovering failures with powerstat, sensors, and dmidecode", "date_modified": "2025-02-19T12:40:58.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/wrong-about-gpu/", "title": "We were wrong about GPUs", "date_modified": "2025-02-14T22:36:31.000Z" }, { "content_html": "Comments", "url": "https://docs.google.com/presentation/d/e/2PACX-1vSmIbSwh1_DXKEMU5YKgYpt5_b4yfOfpfEOKS5_cvtLdiHsX6zt-gNeisamRuCtDtCb2SbTafTI8V47/pub?start=false&loop=false&delayms=3000", "title": "On Bloat", "date_modified": "2025-02-14T07:12:41.000Z" }, { "content_html": "Comments", "url": "https://monzo.com/blog/tolerating-full-cloud-outages-with-monzo-stand-in", "title": "Tolerating full cloud outages with Monzo Stand-in", "date_modified": "2025-02-13T18:23:53.000Z" }, { "content_html": "Comments", "url": "https://elevenreader.io", "title": "ElevenReader by ElevenLabs", "date_modified": "2025-02-12T06:10:25.000Z" }, { "content_html": "Comments", "url": "https://www.runportcullis.co/blog/bulk-data-clickhouse/", "title": "Bulk inserts on ClickHouse: How to avoid overstuffing your instance", "date_modified": "2025-02-11T14:43:45.000Z" }, { "content_html": "Comments", "url": "https://cacm.acm.org/research/metas-hyperscale-infrastructure-overview-and-insights/", "title": "Meta's Hyperscale Infrastructure: Overview and Insights", "date_modified": "2025-02-11T04:19:41.000Z" }, { "content_html": "Comments", "url": "https://github.com/VKCOM/nocc", "title": "Nocc – A Distributed C++ Compiler", "date_modified": "2025-02-11T00:35:34.000Z" }, { "content_html": "Comments", "url": "https://www.docker.com/blog/ga-launch-docker-bake/", "title": "Docker Bake Is Now Generally Available", "date_modified": "2025-02-08T03:39:46.000Z" }, { "content_html": "Comments", "url": "https://www.docker.com/blog/ga-launch-docker-bake/", "title": "Docker Bake is now generally available", "date_modified": "2025-02-08T03:39:46.000Z" }, { "content_html": "Comments", "url": "https://www.cloudflarestatus.com", "title": "Cloudflare R2 Global Outage", "date_modified": "2025-02-06T08:27:37.000Z" }, { "content_html": "

A mirror proxy Google runs on behalf of developers of the Go programming language pushed a backdoored package for more than three years until Monday, after researchers who spotted the malicious code petitioned for it to be taken down twice.

\n

The service, known as the Go Module Mirror, caches open source packages available on GitHub and elsewhere so that downloads are faster and to ensure they are compatible with the rest of the Go ecosystem. By default, when someone uses command-line tools built into Go to download or install packages, requests are routed through the service. A description on the site says the proxy is provided by the Go team and “run by Google.”

\n

Caching in

\n

Since November 2021, the Go Module Mirror has been hosting a backdoored version of a widely used module, security firm Socket said Monday. The file uses “typosquatting,” a technique that gives malicious files names similar to widely used legitimate ones and plants them in popular repositories. In the event someone makes a typo or even a minor variation from the correct name when fetching a file with the command line, they land on the malicious file instead of the one they wanted. (A similar typosquatting scheme is common with domain names, too.)

Read full article

\n

Comments

", "url": "https://arstechnica.com/security/2025/02/backdoored-package-in-go-mirror-site-went-unnoticed-for-3-years/", "title": "Go Module Mirror served backdoor to devs for 3+ years", "date_modified": "2025-02-05T12:25:55.000Z" }, { "content_html": "Comments", "url": "https://github.com/maurobaraldi/terraform-workspaces-aws-multi-account", "title": "Using Terraform Workspace for AWS multi account archtetctures", "date_modified": "2025-02-05T07:36:01.000Z" }, { "content_html": "

Comments

", "url": "https://aws.amazon.com/blogs/migration-and-modernization/deploying-openvms-x86-on-amazon-ec2/", "title": "Deploying OpenVMS x86 on Amazon EC2", "date_modified": "2025-02-05T05:48:06.000Z" }, { "content_html": "Comments", "url": "https://github.com/lsd-so/lsd-mcp", "title": "Show HN: Gave Claude LSD SQL", "date_modified": "2025-02-03T08:11:58.000Z" }, { "content_html": "Comments", "url": "https://www.swift.org/blog/the-next-chapter-in-swift-build-technologies/", "title": "Apple is open sourcing Swift Build", "date_modified": "2025-02-01T16:44:53.000Z" }, { "content_html": "Comments", "url": "https://www.admin-magazine.com/HPC/Articles/Linux-Writecache", "title": "Setting up a Linux writecache as a RAM disk", "date_modified": "2025-02-01T02:25:39.000Z" }, { "content_html": "Comments", "url": "https://www.githubstatus.com", "title": "GitHub Is Down", "date_modified": "2025-01-30T14:29:28.000Z" }, { "content_html": "Comments", "url": "https://www.dbos.dev/blog/what-is-lightweight-durable-execution", "title": "Why Durable Execution Should Be Lightweight", "date_modified": "2025-01-30T14:19:12.000Z" }, { "content_html": "Comments", "url": "https://www.instantdb.com/essays/pg_upgrade", "title": "A Major Postgres Upgrade with Zero Downtime", "date_modified": "2025-01-29T16:57:59.000Z" }, { "content_html": "Comments", "url": "https://github.com/lastmile-ai/mcp-agent", "title": "Show HN: Mcp-Agent – Build effective agents with Model Context Protocol", "date_modified": "2025-01-29T16:26:07.000Z" }, { "content_html": "Comments", "url": "https://www.jvt.me/posts/2025/01/27/go-tools-124/", "title": "Go 1.24's go tool is one of the best additions to the ecosystem in years", "date_modified": "2025-01-27T20:33:43.000Z" }, { "content_html": "Comments", "url": "https://www.ubicloud.com/blog/cloud-virtualization-red-hat-aws-firecracker-and-ubicloud-internals", "title": "Cloud Virtualization: Red Hat, AWS Firecracker, and Ubicloud internals", "date_modified": "2025-01-24T15:59:23.000Z" }, { "content_html": "Comments", "url": "https://blog.railway.com/p/slack-overflow", "title": "How we scaled Slack to support 1000s of developers", "date_modified": "2025-01-24T07:10:25.000Z" }, { "content_html": "Comments", "url": "https://bun.sh/blog/bun-v1.2", "title": "Bun 1.2 Is Released", "date_modified": "2025-01-23T06:50:28.000Z" }, { "content_html": "

Comments

", "url": "https://tailwindcss.com/blog/tailwindcss-v4", "title": "Tailwind CSS v4.0", "date_modified": "2025-01-23T03:24:04.000Z" }, { "content_html": "Comments", "url": "https://www.trae.ai/home", "title": "Trae: An AI-powered IDE by ByteDance", "date_modified": "2025-01-23T01:21:37.000Z" }, { "content_html": "Comments", "url": "https://tailwindcss.com/blog/tailwindcss-v4", "title": "Tailwind CSS v4.0", "date_modified": "2025-01-23T00:33:02.000Z" }, { "content_html": "Comments", "url": "https://ninkovic.dev/blog/2025/think-twice-before-using-github-actions", "title": "I'll think twice before using GitHub Actions again", "date_modified": "2025-01-20T03:41:27.000Z" }, { "content_html": "Comments", "url": "https://www.economist.com/finance-and-economics/2025/01/16/the-traitors-a-reality-tv-show-offers-a-useful-economics-lesson", "title": "“The Traitors”, a reality TV show, offers a useful economics lesson", "date_modified": "2025-01-19T08:56:35.000Z" }, { "content_html": "

Comments

", "url": "https://blog.sourcedive.net/brood-war-korean-translations/", "title": "Brood War Korean Translations", "date_modified": "2025-01-17T17:13:27.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=42738479", "title": "Ask HN: Google forcibly enabled Gemini in our Corp Org. How to disable?", "date_modified": "2025-01-17T15:25:29.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=itpcsQQvgAQ", "title": "Nintendo announces the Switch 2 [video]", "date_modified": "2025-01-16T13:08:14.000Z" }, { "content_html": "

Comments

", "url": "https://www.dgt.is/blog/2025-01-10-nix-death-by-a-thousand-cuts/", "title": "Nix - Death by a thousand cuts", "date_modified": "2025-01-15T00:52:30.000Z" }, { "content_html": "

Microsoft CEO Satya Nadella has announced a dramatic restructuring of the company's engineering organization, which is pivoting the company's focus to developing the tools that will underpin agentic AI.

\n

Dubbed \"CoreAI - Platform and Tools,\" the new division rolls the existing AI platform team and the previous developer division (responsible for everything from .NET to Visual Studio) along with some other teams into one big group.

\n

As for what this group will be doing specifically, it's basically everything that's mission-critical to Microsoft in 2025, as Nadella tells it:

Read full article

\n

Comments

", "url": "https://arstechnica.com/gadgets/2025/01/amid-a-flurry-of-hype-microsoft-reorganizes-entire-dev-team-around-ai/", "title": "Amid a flurry of hype, Microsoft reorganizes entire dev team around AI", "date_modified": "2025-01-14T21:00:07.000Z" }, { "content_html": "Comments", "url": "https://kevinmunger.substack.com/p/in-the-belly-of-the-mrbeast", "title": "In the belly of the MrBeast", "date_modified": "2025-01-14T13:05:31.000Z" }, { "content_html": "Comments", "url": "https://sliplane.io", "title": "Show HN: Simple Docker Hosting", "date_modified": "2025-01-14T12:35:13.000Z" }, { "content_html": "Comments", "url": "https://www.githubstatus.com/incidents/qd96yfgvmcf9", "title": "GitHub Git Operations Are Down", "date_modified": "2025-01-13T23:47:31.000Z" }, { "content_html": "Comments", "url": "http://134.0.119.41", "title": "Disco Elysium Explorer", "date_modified": "2025-01-13T03:11:37.000Z" }, { "content_html": "Comments", "url": "http://muratbuffalo.blogspot.com/2023/08/distributed-transactions-at-scale-in.html", "title": "Distributed Transactions at Scale in Amazon DynamoDB", "date_modified": "2025-01-12T11:12:29.000Z" }, { "content_html": "Comments", "url": "https://www.dgt.is/blog/2025-01-10-nix-death-by-a-thousand-cuts/", "title": "Nix – Death by a Thousand Cuts", "date_modified": "2025-01-11T16:19:56.000Z" }, { "content_html": "Comments", "url": "https://varoa.net/2025/01/09/serverless.html", "title": "Why aren't we all serverless yet?", "date_modified": "2025-01-09T13:12:57.000Z" }, { "content_html": "Comments", "url": "https://miguelcarranza.es/cto-year-7", "title": "Year 7 as a CTO", "date_modified": "2025-01-06T14:27:31.000Z" }, { "content_html": "Comments", "url": "https://futureboy.us/blog/twofifty.html", "title": "Engineer eats efficiently for $2.50 a day (2016)", "date_modified": "2025-01-06T01:56:42.000Z" }, { "content_html": "Comments", "url": "https://blog.mggross.com/intercepting-syscalls/", "title": "Reliable system call interception", "date_modified": "2025-01-05T15:58:05.000Z" }, { "content_html": "Comments", "url": "https://www.texasmonthly.com/food/yemeni-coffee-shops-booming-in-texas/", "title": "Yemeni Coffee Shops in Texas", "date_modified": "2025-01-03T11:11:39.000Z" }, { "content_html": "

Doesn’t link directly to video, but to collection of videos.

\n

Comments

", "url": "https://www.cs.kent.ac.uk/ErlangMasterClasses/", "title": "Erlang master classes", "date_modified": "2025-01-02T15:51:21.000Z" }, { "content_html": "Comments", "url": "https://borretti.me/article/how-i-use-claude", "title": "How I Use Claude", "date_modified": "2025-01-02T05:35:06.000Z" }, { "content_html": "

Andy rises from the ashes of his dead startup and discusses what happened in 2024 in the database game.

", "url": "https://www.cs.cmu.edu/~pavlo/blog/2025/01/2024-databases-retrospective.html", "title": "Databases in 2024: A Year in Review", "date_modified": "2025-01-01T00:32:11.864Z" }, { "content_html": "Comments", "url": "https://hardcoresoftware.learningbyshipping.com/p/225-systems-ideas-that-sound-good", "title": "Systems ideas that sound good but almost never work", "date_modified": "2024-12-31T16:47:54.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=42548480", "title": "Is there such a thing as \"private, interactive databases\" for SaaS's", "date_modified": "2024-12-30T11:49:03.000Z" }, { "content_html": "Comments", "url": "https://blog.flippercloud.io/per-seat-pricing-sucks/", "title": "Per Seat Pricing Sucks", "date_modified": "2024-12-28T16:58:35.000Z" }, { "content_html": "Comments", "url": "https://www.permissionlessbook.com", "title": "Permissionless. A Manifesto for the Future of Everything", "date_modified": "2024-12-28T04:17:44.000Z" }, { "content_html": "Comments", "url": "https://mitchellh.com/writing/ghostty-1-0-reflection", "title": "Ghostty: Reflecting on Reaching 1.0", "date_modified": "2024-12-28T00:16:34.000Z" }, { "content_html": "Comments", "url": "https://ghostty.org/", "title": "Ghostty 1.0", "date_modified": "2024-12-26T20:14:57.000Z" }, { "content_html": "Comments", "url": "https://engineering.prezi.com/how-using-availability-zones-can-eat-up-your-budget-our-journey-from-prometheus-to-be8a816f7efe", "title": "Using AZs can eat up your budget – From Prometheus to VictoriaMetrics", "date_modified": "2024-12-26T08:09:41.000Z" }, { "content_html": "Comments", "url": "https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/", "title": "A Practitioner's Guide to Wide Events", "date_modified": "2024-12-23T20:29:29.000Z" }, { "content_html": "Comments", "url": "https://github.com/fish-shell/fish-shell/releases/tag/4.0b1", "title": "Fish has been ported to Rust", "date_modified": "2024-12-22T16:20:23.000Z" }, { "content_html": "Comments", "url": "https://buildingslack.com/the-death-of-glitch-the-birth-of-slack/", "title": "The death of Glitch, the birth of Slack", "date_modified": "2024-12-22T10:20:39.000Z" }, { "content_html": "Comments", "url": "https://s2.dev/blog/intro", "title": "Introducing S2", "date_modified": "2024-12-21T15:11:19.000Z" }, { "content_html": "Comments", "url": "https://github.com/libriscv/multi_tenant_drogon", "title": "Show HN: Ephemeral VMs in 1 Microsecond", "date_modified": "2024-12-20T10:43:36.000Z" }, { "content_html": "Comments", "url": "https://wordpress.org/news/2024/12/holiday-break/", "title": "Matt Mullenweg temporarily shuts down some Wordpress.org functions", "date_modified": "2024-12-20T10:07:42.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/Articles/1002820/", "title": "Fish shell announces 4.0 release", "date_modified": "2024-12-19T14:36:36.000Z" }, { "content_html": "Comments", "url": "https://go.dev/blog/protobuf-opaque", "title": "Go Protobuf: The New Opaque API", "date_modified": "2024-12-16T20:18:01.000Z" }, { "content_html": "Comments", "url": "https://rootly.com/blog/new-blog-series-rescueops", "title": "Rootly New Blog Series: RescueOps", "date_modified": "2024-12-16T14:17:02.000Z" }, { "content_html": "

Comments

", "url": "https://leebriggs.co.uk/blog/2024/12/10/the-death-of-devrel", "title": "The Death of Developer Relations", "date_modified": "2024-12-15T13:29:59.000Z" }, { "content_html": "Comments", "url": "https://kubespec.dev/", "title": "Show HN: Kubernetes Spec Explorer", "date_modified": "2024-12-12T15:02:14.000Z" }, { "content_html": "

Comments

", "url": "https://earthly.dev/blog/ucacher/", "title": "Ucacher: Speeding up GitHub Actions via syscall instrumentation", "date_modified": "2024-12-11T17:44:11.000Z" }, { "content_html": "Comments", "url": "https://github.com/bazelbuild/bazel/releases/tag/8.0.0", "title": "Bazel 8.0 Released", "date_modified": "2024-12-09T21:33:12.000Z" }, { "content_html": "Comments", "url": "https://airport.revolvertype.com/", "title": "Now Boarding: The Story of Airport", "date_modified": "2024-12-07T10:48:52.000Z" }, { "content_html": "Comments", "url": "https://github.com/jdx/mise", "title": "Mise: Dev tools, env vars, task runner", "date_modified": "2024-12-07T07:21:40.000Z" }, { "content_html": "Comments", "url": "https://terriblesoftware.org/2024/12/04/the-6-mistakes-youre-going-to-make-as-a-new-manager/", "title": "Mistakes You're Going to Make as a New Manager", "date_modified": "2024-12-06T16:56:50.000Z" }, { "content_html": "Comments", "url": "https://www.acton-lang.org/", "title": "The Acton Programming Language", "date_modified": "2024-12-05T21:36:59.000Z" }, { "content_html": "Comments", "url": "https://github.com/facebook/react/blob/main/CHANGELOG.md", "title": "React v19 has been released", "date_modified": "2024-12-05T18:50:16.000Z" }, { "content_html": "Comments", "url": "https://github.com/facebook/react/blob/main/CHANGELOG.md", "title": "React 19", "date_modified": "2024-12-05T18:50:16.000Z" }, { "content_html": "Comments", "url": "https://openai.com/index/introducing-chatgpt-pro/", "title": "ChatGPT Pro", "date_modified": "2024-12-05T18:09:31.000Z" }, { "content_html": "

Today, we are announcing Buy with AWS, a new way to discover and purchase solutions available in AWS Marketplace from AWS Partner sites. You can use Buy with AWS to accelerate and streamline your product procurement process on websites outside of Amazon Web Services (AWS). This feature provides you the ability to find, try, and buy solutions from Partner websites using your AWS account

\n

AWS Marketplace is a curated digital store for you to find, buy, deploy, and manage cloud solutions from Partners. Buy with AWS is another step towards AWS Marketplace making it easy for you to find and procure the right Partner solutions, when and where you need them. You can conveniently find and procure solutions in AWS Marketplace, through integrated AWS service consoles, and now on Partner websites.

\n

Accelerate cloud solution discovery and evaluation

\n

You can now discover solutions from Partners available for purchase through AWS Marketplace as you explore solutions on the web beyond AWS.

\n

Look for products that are “Available in AWS Marketplace” when browsing on Partner sites, then accelerate your evaluation process with fast access to free trials, demo requests, and inquiries for custom pricing.

\n

For example, I want to evaluate Wiz to see how it can help with my cloud security requirements. While browsing the Wiz website, I come across a page where I see “Connect Wiz with Amazon Web Services (AWS)”.

\n

\"Wiz

\n

I choose Try with AWS. It asks me to sign in to my AWS account if I’m not signed in already. I’m then presented with a Wiz and AWS co-branded page for me to sign up for the free trial.

\n

\"Wiz

\n

The discovery experience that you see will vary depending on type of the Partner website you’re shopping from. Wiz is an example of how Buy with AWS can be implemented by an independent software vendor (ISV). Now, let’s look at an example of an AWS Marketplace Channel Partner, or reseller, who operates a storefront of their own.

\n

I browse to the Bytes storefront with product listings from AWS Marketplace. I have the option to filter and search from the curated product listings, which are available in AWS Marketplace, on the Bytes site.

\n

\"Bytes

\n

I choose View Details for Fortinet and see an option to Request Private Offer from AWS.

\n

\"Bytes

\n

As you can tell, on a Channel Partner site, you can browse curated product listings available in AWS Marketplace, filter products, and request custom pricing directly from their website.

\n

Streamline product procurement on AWS Partner sites
I had a seamless experience using Buy with AWS to access a free trial for Wiz and browse through the Bytes storefront to request a private offer.

\n

Now I want to try Databricks for one of the applications I’m building. I sign up for a Databricks trial through their website.

\n

\"Database

\n

I chose Upgrade and see Databricks is available in AWS Marketplace, which gives me the option to Buy with AWS.

\n

\"Option

\n

I choose Buy with AWS, and after I sign in to my AWS account, I land on a Databricks and AWS Marketplace co-branded procurement page.

\n

\"Databricks

\n

I complete the purchase on the co-branded procurement page and continue to set up my Databricks account.

\n

\"Databricks

\n

As you can tell, I didn’t have to navigate the challenge of managing procurement processes for multiple vendors. I also didn’t have to speak with a sales representative or onboard a new vendor in my billing system, which would have required multiple approvals and delayed the overall process.

\n

Access centralized billing and benefits through AWS Marketplace
Because Buy with AWS purchases are transacted through and managed in AWS Marketplace, you also benefit from the post-purchase experience of AWS Marketplace, including consolidated AWS billing, centralized subscription management, and access to cost optimization tools.

\n

For example, through the AWS Billing and Cost Management console, I can centrally manage all my AWS purchases, including Buy with AWS purchases, from one dashboard. I can easily access and process invoices for all of my organization’s AWS purchases. I also need to have valid AWS Identity and Access Management (IAM) permissions to manage subscriptions and make a purchase through AWS Marketplace.

\n

\n

AWS Marketplace not only simplifies my billing but also helps in maintaining governance over spending by helping me manage purchasing authority and subscription access for my organization with centralized visibility and controls. I can manage my budget with pricing flexibility, cost transparency, and AWS cost management tools.

\n

Buy with AWS for Partners
Buy with AWS enables Partners who sell or resell products in AWS Marketplace to create new solution discovery and buying experiences for customers on their own websites. By adding call to action (CTA) buttons to their websites such as “Buy with AWS”, “Try free with AWS”, “Request private offer”, and “Request demo”, Partners can help accelerate product evaluation and the path-to-purchase for customers.

\n

By integrating with AWS Marketplace APIs, Partners can display products from the AWS Marketplace catalog, allow customers to sort and filter products, and streamline private offers. Partners implementing Buy with AWS can access AWS Marketplace creative and messaging resources for guidance on building their own web experiences. Partners who implement Buy with AWS can access metrics for insights into engagement and conversion performance.

\n

The Buy with AWS onboarding guide in the AWS Marketplace Management Portal details how Partners can get started.

\n

Learn more
Visit the Buy with AWS page to learn more and explore Partner sites that offer Buy with AWS.

\n

To learn more about selling or reselling products using Buy with AWS on your website, visit:

\n \n

– Prasad

", "url": "https://aws.amazon.com/blogs/aws/introducing-buy-with-aws-an-accelerated-procurement-experience-on-aws-partner-sites-powered-by-aws-marketplace/", "title": "Introducing Buy with AWS: an accelerated procurement experience on AWS Partner sites, powered by AWS Marketplace", "date_modified": "2024-12-04T23:30:08.000Z" }, { "content_html": "Comments", "url": "https://www.cnn.com/2024/12/04/us/brian-thompson-united-healthcare-death/index.html", "title": "UnitedHealthcare CEO fatally shot in midtown Manhattan", "date_modified": "2024-12-04T14:52:55.000Z" }, { "content_html": "Comments", "url": "https://blog.kerollmops.com/meilisearch-is-too-slow", "title": "Drawbacks and solutions for the Meilisearch document indexer", "date_modified": "2024-12-04T00:21:45.000Z" }, { "content_html": "

Comments

", "url": "https://aws.amazon.com/about-aws/whats-new/2024/12/amazon-s3-tables-apache-iceberg-tables-analytics-workloads/", "title": "Amazon S3 Tables", "date_modified": "2024-12-03T19:47:01.000Z" }, { "content_html": "Comments", "url": "https://aws.amazon.com/rds/aurora/dsql/", "title": "Amazon Aurora DSQL", "date_modified": "2024-12-03T17:30:41.000Z" }, { "content_html": "

AWS customers make use of Amazon Simple Storage Service (Amazon S3) at an incredible scale, regularly creating individual buckets that contain billions or trillions of objects! At that scale, finding the objects which meet particular criteria — objects with keys that match a pattern, objects of a particular size, or objects with a specific tag — becomes challenging. Our customers have had to build systems that capture, store, and query for this information. These systems can become complex and hard to scale, and can fall out of sync with the actual state of the bucket and the objects within.

\n

Rich Metadata
Today we are enabling in preview automatic generation of metadata that is captured when S3 objects are added or modified, and stored in fully managed Apache Iceberg tables. This allows you to use Iceberg-compatible tools such as Amazon Athena, Amazon Redshift, Amazon QuickSight, and Apache Spark to easily and efficiently query the metadata (and find the objects of interest) at any scale. As a result, you can quickly find the data that you need for your analytics, data processing, and AI training workloads.

\n

For video inference responses stored in S3, Amazon Bedrock will annotate the content it generates with metadata that will allow you to identify the content as AI-generated, and to know which model was used to generate it.

\n

The metadata schema contains over 20 elements including the bucket name, object key, creation/modification time, storage class, encryption status, tags, and user metadata. You can also store additional, application-specific descriptive information in a separate table and then join it with the metadata table as part of your query.

\n

How it Works
You can enable capture of rich metadata for any of your S3 buckets by specifying the location (an S3 table bucket and a table name) where you want the metadata to be stored. Capture of updates (object creations, object deletions, and changes to object metadata) begins right away and will be stored in the table within minutes. Each update generates a new row in the table, with a record type (CREATE, UPDATE_METADATA, or DELETE) and a sequence number. You can retrieve the historical record for a given object by running a query that orders the results by sequence number.

\n

Enabling and Querying Metadata
I start by creating a table bucket for my metadata using the create-table-bucket command (this can also be done from the AWS Management Console or with an API call):

\n
\n 
$ aws s3tables create-table-bucket --name jbarr-table-bucket-1 --region us-east-2\n--------------------------------------------------------------------------------\n|                               CreateTableBucket                              |\n+-----+------------------------------------------------------------------------+\n|  arn|  arn:aws:s3tables:us-east-2:123456789012:bucket/jbarr-table-bucket-1   |\n+-----+------------------------------------------------------------------------+\n
\n
\n

Then I specify the table bucket (by ARN) and the desired table name by putting this JSON into a file (I’ll call it config.json):

\n
\n 
{\n  \"S3TablesDestination\": {\n    \"TableBucketArn\": \"arn:aws:s3tables:us-east-2:123456789012:bucket/jbarr-table-bucket-1\",\n    \"TableName\": \"jbarr_data_bucket_1_table\"\n  }\n}
\n
\n

And then I attach this configuration to my data bucket (the one that I want to capture metadata for):

\n
\n 
$ aws s3api create-bucket-metadata-table-configuration \\\n  --bucket jbarr-data-bucket-1 \\\n  --metadata-table-configuration file://./config.json \\\n  --region us-east-2
\n
\n

For testing purposes I installed Apache Spark on an EC2 instance and after a little bit of setup I was able to run queries by referencing the Amazon S3 Tables Catalog for Apache Iceberg package and adding the metadata table (as mytablebucket) to the command line:

\n
\n 
$ bin/spark-shell \\\n--packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.6.0 \\\n--jars ~/S3TablesCatalog.jar \\\n--master yarn \\\n--conf \"spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions\" \\\n--conf \"spark.sql.catalog.mytablebucket=org.apache.iceberg.spark.SparkCatalog\" \\\n--conf \"spark.sql.catalog.mytablebucket.catalog-impl=com.amazon.s3tables.iceberg.S3TablesCatalog\" \\\n--conf \"spark.sql.catalog.mytablebucket.warehouse=arn:aws:s3tables:us-east-2:123456789012:bucket/jbarr-table-bucket-1\"
\n
\n

Here is the current schema for the Iceberg table:

\n
\n 
scala> spark.sql(\"describe table mytablebucket.aws_s3_metadata.jbarr_data_bucket_1_table\").show(100,35)\n\n+---------------------+------------------+-----------------------------------+\n|             col_name|         data_type|                            comment|\n+---------------------+------------------+-----------------------------------+\n|               bucket|            string|   The general purpose bucket name.|\n|                  key|            string|The object key name (or key) tha...|\n|      sequence_number|            string|The sequence number, which is an...|\n|          record_type|            string|The type of this record, one of ...|\n|     record_timestamp|     timestamp_ntz|The timestamp that's associated ...|\n|           version_id|            string|The object's version ID. When yo...|\n|     is_delete_marker|           boolean|The object's delete marker statu...|\n|                 size|            bigint|The object size in bytes, not in...|\n|   last_modified_date|     timestamp_ntz|The object creation date or the ...|\n|                e_tag|            string|The entity tag (ETag), which is ...|\n|        storage_class|            string|The storage class that's used fo...|\n|         is_multipart|           boolean|The object's upload type. If the...|\n|    encryption_status|            string|The object's server-side encrypt...|\n|is_bucket_key_enabled|           boolean|The object's S3 Bucket Key enabl...|\n|          kms_key_arn|            string|The Amazon Resource Name (ARN) f...|\n|   checksum_algorithm|            string|The algorithm that's used to cre...|\n|          object_tags|map<string,string>|The object tags that are associa...|\n|        user_metadata|map<string,string>|The user metadata that's associa...|\n|            requester|            string|The AWS account ID of the reques...|\n|    source_ip_address|            string|The source IP address of the req...|\n|           request_id|            string|The request ID. For records that...|\n+---------------------+------------------+-----------------------------------+\n
\n
\n

Here’s a simple query that shows some of the metadata for the ten most recent updates:

\n
\n 
scala> spark.sql(\"SELECT key,size, storage_class,encryption_status \\\n  FROM mytablebucket.aws_s3_metadata.jbarr_data_bucket_1_table \\\n  order by last_modified_date DESC LIMIT 10\").show(false)\n+--------------------+------+-------------+-----------------+                   \n|key                 |size  |storage_class|encryption_status|\n+--------------------+------+-------------+-----------------+\n|wnt_itco_2.png      |36923 |STANDARD     |SSE-S3           |\n|wnt_itco_1.png      |37274 |STANDARD     |SSE-S3           |\n|wnt_imp_new_1.png   |15361 |STANDARD     |SSE-S3           |\n|wnt_imp_change_3.png|67639 |STANDARD     |SSE-S3           |\n|wnt_imp_change_2.png|67639 |STANDARD     |SSE-S3           |\n|wnt_imp_change_1.png|71182 |STANDARD     |SSE-S3           |\n|wnt_email_top_4.png |135164|STANDARD     |SSE-S3           |\n|wnt_email_top_2.png |117171|STANDARD     |SSE-S3           |\n|wnt_email_top_3.png |55913 |STANDARD     |SSE-S3           |\n|wnt_email_top_1.png |140937|STANDARD     |SSE-S3           |\n+--------------------+------+-------------+-----------------+
\n
\n

In a real-world situation I would query the table using one of the AWS or open source analytics tools that I mentioned earlier.

\n

Console Access
I can also set up and manage the metadata configuration for my buckets using the Amazon S3 Console by clicking the Metadata tab:

\n

\n

Available Now
Amazon S3 Metadata is available in preview now and you can start using it today in the US East (Ohio, N. Virginia) and US West (Oregon) AWS Regions.

\n

Integration with AWS Glue Data Catalog is in preview, allowing you to query and visualize data—including S3 Metadata tables—using AWS Analytics services such as Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon QuickSight.

\n

Pricing is based on the number updates (object creations, object deletions, and changes to object metadata) with an additional charge for storage of the metadata table. For more pricing information, visit the S3 Pricing page.

\n

I’m confident that you will be able to make use of this metadata in many powerful ways, and am looking forward to hearing about your use cases. Let me know what you think!

\n

\n

— Jeff;

\n

", "url": "https://aws.amazon.com/blogs/aws/introducing-queryable-object-metadata-for-amazon-s3-buckets-preview/", "title": "Introducing queryable object metadata for Amazon S3 buckets (preview)", "date_modified": "2024-12-03T16:47:12.000Z" }, { "content_html": "

Comments

", "url": "https://www.macchaffee.com/blog/2024/the-next-platform/", "title": "The next platform", "date_modified": "2024-12-03T02:52:29.000Z" }, { "content_html": "Comments", "url": "https://modal.com/blog/vprox", "title": "Static IPs for Serverless Containers", "date_modified": "2024-12-02T20:04:29.000Z" }, { "content_html": "

AWS re:Invent 2024, our flagship an\"Imagenual conference, is taking place Dec. 2-6, 2024, in Las Vegas. This premier cloud computing event brings together the global cloud computing community for a week of keynotes, technical sessions, product launches, and networking opportunities. As AWS continues to unveil its latest innovations and services throughout the conference, we’ll keep you updated here with all the major product announcements.

\n

Additional re:Invent resources:

\n \n

(This post was last updated:  8:26 a.m. PST, Dec. 3, 2024.)

\n
\n

Quick category links:

\n

Analytics | Application Integration | Business Applications | Compute | Containers | Database | Generative AI / Machine Learning | Management & Governance | Migration & Transfer Services | Security, Identity, & Compliance | Storage

\n
\n

Analytics

\n

AWS Clean Rooms now supports multiple clouds and data sources
With expanded data sources, AWS Clean Rooms helps customers securely collaborate with their partners’ data across clouds, eliminating data movement, safeguarding sensitive information, promoting data freshness, and streamlining cross-company insights.

\n
\n

Application Integration

\n
\n Securely share AWS resources across VPC and account boundaries with PrivateLink, VPC Lattice, EventBridge, and Step Functions\n
Orchestrate hybrid workflows accessing private HTTPS endpoints – no more Lambda/SQS workarounds. EventBridge and Step Functions natively support private resources, simplifying cloud modernization.\n
\n
 \n

\n

Business Applications

\n

Newly enhanced Amazon Connect adds generative AI, WhatsApp Business, and secure data collection
Use innovative tools like generative AI for segmentation and campaigns, WhatsApp Business, data privacy controls for chat, AI guardrails, conversational AI bot management, and enhanced analytics to elevate customer experiences securely and efficiently.

\n
\n

Compute

\n

Amazon EC2 Trn2 Instances and Trn2 UltraServers for AI/ML training and inference are now available
With 4x faster speed, 4x more memory bandwidth, 3x higher memory capacity than predecessors, and 30% higher floating-point operations, these instances deliver unprecedented compute power for ML training and gen AI.

\n

New Amazon EC2 P5en instances with NVIDIA H200 Tensor Core GPUs and EFAv3 networking
Amazon EC2 P5en instances deliver up to 3,200 Gbps network bandwidth with EFAv3 for accelerating deep learning, generative AI, and HPC workloads with unmatched efficiency.

\n

Introducing storage optimized Amazon EC2 I8g instances powered by AWS Graviton4 processors and 3rd gen AWS Nitro SSDs
Elevate storage performance with AWS’s newest I8g instances, which deliver unparalleled speed and efficiency for I/O-intensive workloads.

\n

Now available: Storage optimized Amazon EC2 I7ie instances
New AWS I7ie instances deliver unbeatable storage performance: up to 120TB NVMe, 40% better compute performance and up to 65% better real-time storage performance.

\n
\n

Containers

\n

Use your on-premises infrastructure in Amazon EKS clusters with Amazon EKS Hybrid Nodes
Unify Kubernetes management across your cloud and on-premises environments with Amazon EKS Hybrid Nodes – use existing hardware while offloading control plane responsibilities to EKS for consistent operations.

\n

Streamline Kubernetes cluster management with new Amazon EKS Auto Mode
With EKS Auto Mode, AWS simplifies Kubernetes cluster management, automating compute, storage, and networking, enabling higher agility and performance while reducing operational overhead.

\n
\n

Database

\n

Amazon MemoryDB Multi-Region is now generally available
Build highly available, globally distributed apps with microsecond latencies across Regions, automatic conflict resolution, and up to 99.999% availability.

\n
\n

Generative AI / Machine Learning

\n

New RAG evaluation and LLM-as-a-judge capabilities in Amazon Bedrock
Evaluate AI models and applications efficiently with Amazon Bedrock’s new LLM-as-a-judge capability for model evaluation and RAG evaluation for Knowledge Bases, offering a variety of quality and responsible AI metrics at scale.

\n

Enhance your productivity with new extensions and integrations in Amazon Q Business
Seamlessly access AI assistance within work applications with Amazon Q Business’s new browser extensions and integrations.

\n

New APIs in Amazon Bedrock to enhance RAG applications, now available
With custom connectors and reranking models, you can enhance RAG applications by enabling direct ingestion to knowledge bases without requiring a full sync, and improving response relevance through advanced reranking models.

\n

Introducing new PartyRock capabilities and free daily usage
Unleash your creativity with PartyRock’s new AI capabilities: generate images, analyze visuals, search hundreds of thousands of apps, and process multiple docs simultaneously – no coding required.

\n
\n Amazon Q Business adds support to extract insights from visual elements within documents\n
\n
\n Users can now query information embedded in various types of visuals, including diagrams, infographics, charts, and other image-based content.\n
\n

Management & Governance

\n

Container Insights with enhanced observability now available in Amazon ECS
With granular visibility into container workloads, CloudWatch Container Insights with enhanced observability for Amazon ECS enables proactive monitoring and faster troubleshooting, enhancing observability and improving application performance.

\n

New Amazon CloudWatch Database Insights: Comprehensive database observability from fleets to instances
Monitor Amazon Aurora databases and gain comprehensive visibility into MySQL and PostgreSQL fleets and instances, analyze performance bottlenecks, track slow queries, set SLOs, and explore rich telemetry.

\n

New Amazon CloudWatch and Amazon OpenSearch Service launch an integrated analytics experience
Unlock out-of-the-box OpenSearch dashboards and two additional query languages, OpenSearch SQL and PPL, for analyzing CloudWatch logs. OpenSearch customers can now analyze CloudWatch Logs without having to duplicate data.

\n
\n

Migration & Transfer Services

\n

AWS Database Migration Service now automates time-intensive schema conversion tasks using generative AI
AWS DMS Schema Conversion converts up to 90% of your schema to accelerate your database migrations and reduce manual effort with the power of generative AI.

\n

Announcing AWS Transfer Family web apps for fully managed Amazon S3 file transfers
AWS Transfer Family web apps are a new resource that you can use to create a simple interface for authorized line-of-business users to access data in Amazon S3 through a customizable web browser.

\n

Introducing default data integrity protections for new objects in Amazon S3
Amazon S3 updates the default behavior of object upload requests with new data integrity protections that build upon S3’s existing durability posture.

\n
\n

Security, Identity, & Compliance

\n

New AWS Security Incident Response helps organizations respond to and recover from security events
AWS introduces a new service to streamline security event response, providing automated triage, coordinated communication, and expert guidance to recover from cybersecurity threats.

\n

Introducing Amazon GuardDuty Extended Threat Detection: AI/ML attack sequence identification for enhanced cloud security
AWS extends GuardDuty with AI/ML capabilities to detect complex attack sequences across workloads, applications, and data, correlating multiple security signals over time for proactive cloud security.

\n

Simplify governance with declarative policies
With only a few steps, create declarative policies and enforce desired configuration for AWS services across your organization, reducing ongoing governance overhead and providing transparency for administrators and end users.

\n

AWS Verified Access now supports secure access to resources over non-HTTP(S) protocols (preview)
With only a few steps, create declarative policies and enforce desired configuration for AWS services across your organization, reducing ongoing governance overhead and providing transparency for administrators and end users.

\n

Introducing Amazon OpenSearch Service and Amazon Security Lake integration to simplify security analytics
Analyze security logs without data duplication; Amazon OpenSearch Service now offers zero-ETL integration with Amazon Security Lake for efficient threat hunting and investigations.

\n
\n

Storage

\n

Announcing Amazon FSx Intelligent-Tiering, a new storage class for FSx for OpenZFS
Delivering NAS capabilities with automatic data tiering among frequently accessed, infrequent, and archival storage tiers, Amazon FSx Intelligent-Tiering offers high performance up to 400K IOPS, 20 GB/s throughput, seamless integration with AWS services.

\n

New physical AWS Data Transfer Terminals let you upload to the cloud faster
Rapidly upload large datasets to AWS at blazing speeds with the new AWS Data Transfer Terminal, secure physical locations offering high throughput connection.

\n

Connect users to data through your apps with Storage Browser for Amazon S3
Storage Browser for Amazon S3 is an open source interface component that you can add to your web applications to provide your authorized end users, such as customers, partners, and employees, with access to easily browse, upload, download, copy, and delete data in S3.

", "url": "https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2024/", "title": "Top announcements of AWS re:Invent 2024", "date_modified": "2024-12-02T05:06:42.000Z" }, { "content_html": "

Last year, we announced enhanced observability in Amazon CloudWatch Container Insights, a new capability to improve your observability for Amazon Elastic Kubernetes Service (Amazon EKS). This capability helps you detect and fix container issues faster by providing detailed performance metrics and logs.

\n

Expanding this capability, today we’re launching enhanced observability for your container workloads running on Amazon Elastic Container Service (Amazon ECS). This new capability will help reduce your mean time to detect (MTTD) and mean time to repair (MTTR) for your overall applications, helping prevent issues that could negatively impact your user experience.

\n

Here’s a quick look at Container Insights with enhanced observability for Amazon ECS.

\n

\n

Container Insights with enhanced observability addresses a critical gap in container monitoring. Previously, correlating metrics with logs and events was a time-consuming process, often requiring manual searches and expertise in application architecture. Now, with this capability, CloudWatch and Amazon ECS automatically collect granular performance metrics such as CPU utilization at both the task and container levels while providing visual drill downs enabling easy root-cause analysis.

\n

This new capability enables the following use cases:

\n \n

Using container insights with enhanced observability for Amazon ECS
There are two ways to enable Container Insights with enhanced observability:

\n
    \n
  1. Cluster-level onboarding – You can enable it for specific clusters individually.
    2. \n
  2. Account-level onboarding – You can also enable it at the account level, which automatically enables observability for all new clusters created in your account. This approach saves time and effort by eliminating the need to manually enable it for each new cluster.
    3. \n
\n

To enable this feature at the account level, I navigate to the Amazon ECS console and select Account settings. Under the CloudWatch Container Insights observability section, I can see it’s currently disabled. I choose Update.

\n

\n

On this page, I find a new option called Container Insights with enhanced observability. I select this option and then choose Save changes.

\n

\n

If I need to enable this capability at the cluster level, I can do so when creating a new cluster.

\n

\n

I can also enable this capability for my existing clusters. To do so, I select Update cluster, and then choose the option.

\n

\n

Once enabled, I can see task-level metrics by navigating to the Metrics tab in my cluster overview console. To access health and performance metrics across my clusters, I can select View Container Insights, which will redirect me to the Container Insights page.

\n

\n

To get a big picture of all my workloads across different clusters, I can navigate to Amazon CloudWatch and then to Container Insights.

\n

\n

This view addresses the challenge of effectively monitoring clusters, services, tasks, and containers by providing a honeycomb visualization that offers an intuitive, high-level summary of cluster health. The dashboard employs a dual-state monitoring approach:

\n
    \n
  1. Alarm state (red or green) – Reflects customer-defined thresholds and alerts, allowing teams to configure monitoring based on their specific requirements
    2. \n
  2. Utilization state (dark blue or light blue) – Uses CloudWatch built-in best practices to monitor resource usage patterns across containers. The darker blue indicates clusters operating under higher utilization, enabling teams to proactively identify potential resource constraints before they impact performance
    3. \n
\n

Let’s say there’s an issue in one of my clusters. I can hover over the cluster to display all the alarms created under that cluster at different layers, from the cluster layer down to the container layer.

\n

\n

I also have the option to view all clusters in a list format. The list format is essential for cross-account observability, displaying account IDs and labels for cluster ownership. This helps DevOps engineers quickly identify and collaborate with account owners to resolve potential application issues.

\n

\n

Now, I’d like to explore further. I select my cluster link, which redirects me to the Container Insights detailed dashboard view. Here, I can see a spike in memory utilization for this cluster.

\n

\n

I can dive deeper into container-level details, which help me quickly identify which services are causing this issue.

\n

\n

Another useful feature I found is the Filters option, which helps me conduct more thorough investigations across containers, services, or tasks in this cluster.

\n

\n

If I need to delve deeper into the application logs to understand the root cause of this issue, I can select the task, choose Actions, and choose which logs I would like to view.

\n

On top of using AWS X-Ray traces, I can investigate another two types of logs here. First, I can use performance logs—structured logs containing metric data—to drill down and identify container-level root causes. Second, I examine collected application or container logs . These logs give me detailed insights into application behavior within the container, helping me trace the sequence of events that led to any issues.

\n

In this case, I use application logs.

\n

\n

This streamlines my journey to troubleshoot my application. In this case, the issue is on the downstream calls to third-party applications, which return timeouts.

\n

\n

This enhanced capability also works with Amazon CloudWatch Application Signals to automatically instrument my application. I can monitor current application health and track long-term application performance against service-level objectives.

\n

I select the Application Signals tab.

\n

\n

This integration with Amazon CloudWatch Application Signals provides me with end-to-end visibility, helping me correlate container performance with end-user experience.

\n

When I select datapoints in the graphs, I can see associated traces, which show me all correlated services and their impact. I can also access relevant logs to understand root causes.

\n

\n

Additional things to know
Here are a couple of important points to note:

\n \n

Get started today and experience improved observability for your container workloads. Learn more on the Amazon CloudWatch documentation page.

\n

Happy monitoring,
— Donnie Prakoso

", "url": "https://aws.amazon.com/blogs/aws/container-insights-with-enhanced-observability-now-available-in-amazon-ecs/", "title": "Container Insights with enhanced observability now available in Amazon ECS", "date_modified": "2024-12-02T03:28:56.000Z" }, { "content_html": "

When I speak to customers who are planning to migrate massive amounts of on-premises data to AWS, they tell me that they want to simplify their storage management, reduce their costs, and to make the data more accessible so that it can be used for analytics, machine learning training, genomics, and other use cases. Customers are already using Network Attached Storage (NAS) on-premises, and are looking for a cloud-based upgrade that offers similar capabilities including point-in-time snapshots, data clones, and user management.

\n

AWS customers such as Amdocs, Vela Games, and Astera Labs have been running their mission-critical and performance-intensive NAS workloads like databases, game development and streaming, and semiconductor chip design on Amazon FSx for OpenZFS. They’ve been using the existing SSD storage class on FSx to provide the predictable, high performance these workloads need. However, many other customers have large data sets that are stored on HDD-based or hybrid SSD/HDD-based NAS storage on prem that find it cost-prohibitive to move their data sets to all-SSD storage. Additionally, these customers are finding it increasingly challenging and expensive to manage provisioned storage on prem for unpredictable data sets and avoid running out of space. And they are keeping their NAS data around for longer because it could have future value for building their next model, investment strategy, or product, but that means they need to spend more time and effort monitoring access patterns and moving data around between hot and cold storage media to optimize costs.

\n

FSx Intelligent-Tiering
Taking all of this into account, I am happy to be able to tell you about the new Amazon FSx Intelligent-Tiering storage class, available today for use with Amazon FSx for OpenZFS file systems. The new storage class is priced 85% lower than the existing SSD storage class and 20% lower than traditional HDD-based deployments on premises, and brings full elasticity and intelligent tiering to NAS data sets.

\n

Your data moves between three storage tiers (Frequent Access, Infrequent Access, and Archive) with no effort on your part, so you get automatic cost savings with no upfront costs or commitments. Here’s how the tiers work:

\n

Frequent Access – Data that has been accessed within the last 30 days is stored in this tier.

\n

Infrequent Access – Data that has been not been accessed for 30 to 90 days is stored in tier, at a 44% cost reduction from Frequent Access.

\n

Archive – Data that has not been accessed for 90 or more days is stored in this tier, at a 65% cost reduction from Infrequent Access.

\n

Regardless of the storage tier, your data is stored across multiple AWS Availability Zones (AZs) for redundancy and availability, and can be retrieved instantly in milliseconds.

\n

There’s no need to manage or pre-provision storage, making this storage class a great fit for uses case such as genomics, financial data analytics, seismic imagery analysis, and machine learning where storage requirements can change dramatically over the course of days or weeks.

\n

Along with the potential for cost savings, you get high performance: up to 400K IOPS and 20 GB/second of throughput for each OpenZFS file system, with a time-to-first-byte of tens of milliseconds for all data, regardless of storage class. You can also configure an SSD-based read cache (64 GiB to 512 TiB) to reduce the time-to-first-byte by 10x to 100x for cached data.

\n

Creating a File System
I can create a file system using the AWS Management Console, CLI, API, or a AWS CloudFormation. From the Console I click Create file system to get started:

\n

\n

I choose Amazon FSx for OpenZFS and click Next:

\n

\n

Then I enter a name (jeff_fsx_openzfs_1) for my file system and select the Intelligent-Tiering storage class. I choose the desired Throughput capacity, and I select one of the three sizing mode options for the read cache, click Next, and confirm my choices in order to create my file system:

\n

\n

It is ready within minutes, and I can NFS mount it to my EC2 instance:

\n
\n 
$ sudo mkdir /fsx_zfs\n$ sudo mount -t nfs -o noatime,nfsvers=4.2,sync,nconnect=16,rsize=1048576,wsize=1048576 \\\n  fs-00fc74f020d1e6f4e.fsx.us-east-2.aws.internal:/fsx/ /fsx_zfs/
\n
\n

After I run a representative workload for a while I can look at the metrics and review the performance of my file system:

\n

\n

It appears that I have plenty of throughput, but my read cache may be larger than needed. I created it in Automatically Provisioned mode, which allocated 3200 GiB of cache. I can change that (and save some money) with a couple of clicks:

\n

\n

I can also change the throughput capacity as needed:

\n

\n

Amazon FSx NAS Features and Attributes
Let’s take a quick look at some of the features which make FSx for OpenZFS and the FSx Intelligent-Tiering storage class a great for for your NAS-level storage needs:

\n

Built-in Backups – Amazon FSx automatically makes a daily backup of each file system during a specified backup window and retains them for a specified retention period. The backups are file-system consistent, highly durable, and incremental. You can also create backups on your own and retain them for as long as needed.

\n

Point-In-Time Snapshots -You can create a read-only image of an OpenZFS volume at any time. The snapshots are stored within the file system and consume storage; they can be used to restore a volume, restore individual files and folders, or to create a new volume as either a clone or a full-copy.

\n

Replication – You can replicate a point-in-time view of an OpenZFS volume to another volume across file systems, AWS Regions, and AWS accounts. FSx uses ZFS send/receive technology behind the scenes to perform this replication and automatically establishes and maintains network connectivity between file systems to handle interruptions and resume data transfer as needed.

\n

Data Compression – You can enable ZSTD or LZ4 compression on your OpenZFS volumes to reduce storage cost and speed up data transfer.

\n

User and Volume Quotas – You can limit the amount of storage consumed by an individual volume or user.

\n

Things to Know
Here are a couple of things to keep in mind before we wrap up:

\n

Regions – This new storage class is available in the US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Mumbai, Singapore, Sydney, Tokyo), Canada (Central), and Europe (Frankfurt, Ireland) AWS Regions.

\n

Pricing – Pricing is based on the amount of primary storage consumed (GB/Month) and read cache provisioned (GB/Month). See the Amazon FSx for OpenZFS Pricing page for more information.

\n

\n

— Jeff;

\n

", "url": "https://aws.amazon.com/blogs/aws/announcing-amazon-fsx-intelligent-tiering-a-new-storage-class-for-fsx-for-openzfs/", "title": "Announcing Amazon FSx Intelligent-Tiering, a new storage class for FSx for OpenZFS", "date_modified": "2024-12-02T03:22:42.000Z" }, { "content_html": "

Update on December 2, 2024: Updated SDKs with default integrity protections will be available in the coming weeks.

\n
\n

At Amazon Web Services (AWS), the vast majority of new capabilities are driven by your direct feedback. Two years ago, Jeff announced additional checksum algorithms and the optional client-side computation of checksums to make sure the objects stored on Amazon S3 are exactly what you sent. You told us you love this extra verification because it gives you confidence the object stored is the one you sent. You also told us you would prefer to have this extra verification enabled automatically, freeing you from developing additional code.

\n

Starting today, we’re updating the Amazon Simple Storage Service (Amazon S3) default behavior when you upload objects. To build upon its existing durability posture, Amazon S3 now automatically verifies that your data is correctly transmitted over the network from your applications to your S3 bucket.

\n

Amazon S3 is designed for 99.999999999% data durability (that’s 11 nines). Amazon S3 has always verified the integrity of object uploads by calculating checksums when objects reach our servers, before they are written to multiple storage devices. Once your data is stored in Amazon S3, it continually monitors data durability over time with periodic integrity checks of data at rest. Amazon S3 also actively monitors the redundancy of your data to help verify that your objects can tolerate the concurrent failure of multiple storage devices.

\n

But data can still face integrity risks as it traverses the public internet before reaching our servers. Issues such as faulty hardware on networks we don’t manage or client software bugs could potentially corrupt or drop data before Amazon S3 has a chance to validate it. Previously, you could extend the integrity protection by providing your own precomputed checksums with your PutObject or UploadPart requests. However, this requires configuring tools and applications to generate and track checksums, which can be complex to implement consistently across all your client applications uploading objects to Amazon S3.

\n

The new default behavior builds upon existing data integrity protections without requiring any changes to your applications. Additionally, the new checksums are stored in the object’s metadata, making them accessible for integrity verification at any time.

\n

Automatic client-side integrity protection
Amazon S3 now extends data integrity protection all the way to client-side applications by default. The latest versions of our AWS SDKs automatically calculate a cyclic redundancy check (CRC)-based checksum for each upload and send it to Amazon S3. Amazon S3 independently calculates a checksum on the server side and validates it against the provided value before durably storing the object and its checksum in the object’s metadata.

\n

When your client application doesn’t send a CRC checksum (maybe it uses an old version of our SDK or you haven’t updated your application custom code yet), Amazon S3 computes a CRC-based checksum anyway and stores it in the object metadata for future reference. You can compare at a later stage the stored CRC with a CRC computed on your side and verify the network transmission was correct.

\n

This new capability provides you with an automatic checksum calculation and validation for new uploads from the latest versions of the AWS SDKs, the AWS Command Line Interface (AWS CLI), and the AWS Management Console. You can also verify the checksum stored in the object’s metadata at any time. The new default data integrity protections use the existing CRC32 and CRC32C algorithms or the new CRC64NVME algorithm. Amazon S3 also provides developers with consistent full-object checksums across single-part and multipart uploads.

\n

When uploading files in multiple parts, the SDKs calculate checksums for each part. Amazon S3 uses these checksums to verify the integrity of each part through the UploadPart API. Additionally, S3 validates the entire file’s size and checksum when you call the CompleteMultipartUpload API.

\n

The CreateMultiPartUpload API introduces a new HTTP header, x-amz-checksum-type, which lets you specify the type of checksum to use. You can choose either a full object checksum (calculated by combining the checksums of all individual parts) or a composite checksum.

\n

The full object checksum is stored with the object metadata for future reference. This new protection works seamlessly with server-side encryption. The consistent behavior across uploads, multipart uploads, downloads, and encryption modes simplifies client-side integrity checks. The ability to use full-object checksums to validate integrity and store them for use later can help you streamline your applications.

\n

Let’s see it in action
To start using this additional integrity protection, update to the latest version of the AWS SDK or AWS CLI. No code changes are required to enable the new integrity protections.

\n

Case 1: Amazon S3 now attaches a checksum to objects on the server side when objects are uploaded without a checksum

\n

I wrote a simple Python script to upload and download content to and from an S3 bucket. I enabled maximum logging verbosity to see the actual HTTP headers sent to and from Amazon S3.

\n
import boto3\nimport logging\n\nBUCKET_NAME=\"aws-news-blog-20241111\"\nCONTENT='Hello World!'\nOBJECT_NAME='test.txt'\n\n# Enable debug logging for boto3 and botocore to stdout (this is verbose !!!)\nlogging.basicConfig(level=logging.DEBUG)\n\n# create a s3 client\nclient = boto3.client('s3')\n\n# put an object\nclient.put_object(Bucket=BUCKET_NAME, Key=OBJECT_NAME, Body=CONTENT)\n\n# get the object \nresponse = client.get_object(Bucket=BUCKET_NAME, Key=OBJECT_NAME)\nprint(response['Body'].read().decode('utf-8'))
\n

In the first step of this demo, I use an old AWS SDK for Python that doesn’t compute the CRC checksum on the client side. Despite this, I can observe that Amazon S3 now responds with a checksum it computed upon receiving the object.

\n
S3 RESPONSE:\n{\n    ...\n    \"x-amz-checksum-crc64nvme\": \"AuUcyF784aU=\",\n    \"x-amz-checksum-type\": \"FULL_OBJECT\",\n    ...\n}
\n

Case 2: Upload with manually pre-computed CRC64NVME checksum, a new checksum type

\n

When I don’t have the option to use the latest version of the AWS SDK, or when I use my own code to upload objects to S3 buckets, I can compute the checksum and send it in the PutObject API request. Here is how I compute the checksum on my content before sending it to Amazon S3. To keep this code short, I use the checksums package available in the new AWS SDK for Python.

\n
from awscrt import checksums\nimport base64\n\nchecksum = checksums.crc64nvme(\"Hello World!\")\nchecksum_bytes = checksum.to_bytes(8, byteorder='big')  # CRC64 is 8 bytes\nchecksum_base64 = base64.b64encode(checksum_bytes)\nprint(checksum_base64)
\n

And when I run it, I see the CRC64NVME checksum is the same as the one returned by Amazon S3 in the previous step.

\n
$ python crc.py\nb'AuUcyF784aU='
\n

I can provide this checksum as part of the PutObject API call.

\n
response = s3.put_object(\n    Bucket=BUCKET_NAME,\n    Key=OBJECT_NAME,\n    Body=b'Hello World!',\n    ChecksumAlgorithm='CRC64NVME', \n    ChecksumCRC64NVME=checksum_base64\n)
\n

Case 3: The new SDKs compute the checksum on the client-side

\n

Now, I run the upload and download script again. This time, I use the latest version of the AWS SDK for Python. I observe that the SDK now sends the CRC headers in the request. The response also contains the checksum. I can easily compare the versions in the request and in the response to make sure the object received is the one I sent.

\n
REQUEST:\n{\n    ...\n    \"x-amz-checksum-crc64nvme\": \"AuUcyF784aU=\",\n    \"x-amz-checksum-type\": \"FULL_OBJECT\",\n    ... \n}
\n

At any time, I can request the object checksum to verify the integrity of my local copy using the HeadObject or GetObject APIs.

\n
 get_response = s3.get_object(\n        Bucket=BUCKET_NAME,\n        Key=OBJECT_NAME,\n        ChecksumMode='ENABLED'\n    )
\n

The response object contains the checksum in the HTTPHeaders field.

\n
{\n...\n    \"x-amz-checksum-crc64nvme\": \"AuUcyF784aU=\",\n    \"x-amz-checksum-type\": \"FULL_OBJECT\",\n...\n}
\n

Case 4: Multi-part uploads with new CRC-based whole-object checksum

\n

When uploading large objects using the CreateMultipartUpload, UploadPart, and CompleteMultipartUpload APIs, the latest version of the SDK will automatically compute the checksums for you.

\n

If you want to validate the integrity of your data by using a known content checksum, you can pre-compute the CRC-based whole-object checksum for multi-part uploads to simplify your client side tooling. When using full object checksums for multi-part uploads, you no longer have to keep track of part level checksums as you upload objects.

\n
\n# precomputed CRC64NVME checksum for the full object\nfull_object_crc64_nvme_checksum = 'Naz0uXkYBPM='\n\n# start multipart upload\ncreate_response = s3.create_multipart_upload(\n            Bucket=BUCKET_NAME,\n            Key=OBJECT_NAME,\n            ChecksumAlgorithm='CRC64NVME',\n            ChecksumType='FULL_OBJECT'\n        )\nupload_id = create_response['UploadId']\n\n# Upload parts\nuploaded_parts = []\n\n# part 1\ndata_part_1 = b'0' * (5 * 1024 * 1024) # minimum part size\nupload_part_response_1 = s3.upload_part(\n    Body=data_part_1,\n    Bucket=BUCKET_NAME,\n    Key=OBJECT_NAME,\n    PartNumber=1,\n    UploadId=upload_id,\n    ChecksumAlgorithm='CRC64NVME'\n)\nuploaded_parts.append({'PartNumber': 1, 'ETag': upload_part_response_1['ETag']})\n\n# part 2\ndata_part_2 = b'0' * (5 * 1024 * 1024)\nupload_part_response_2 = s3.upload_part(\n    Body=data_part_2,\n    Bucket=BUCKET_NAME,\n    Key=OBJECT_NAME,\n    PartNumber=2,\n    UploadId=upload_id,\n    ChecksumAlgorithm='CRC64NVME'\n)\nuploaded_parts.append({'PartNumber': 2, 'ETag': upload_part_response_2['ETag']})\n\n# Complete the multipart upload with the FULL_OBJECT CRC64NVME checksum to validate the integrity of your entire object. \ncomplete_response = s3.complete_multipart_upload(\n            Bucket=BUCKET_NAME,\n            Key=OBJECT_NAME,\n            UploadId=upload_id,\n            ChecksumCRC64NVME=full_object_crc64_nvme_checksum,\n            ChecksumType='FULL_OBJECT',\n            MultipartUpload={'Parts': uploaded_parts}\n        )\nprint(complete_response)
\n

Things to know
For your existing objects, the checksum will be added when you copy them. We updated the CopyObject API so you can choose the desired checksum algorithm for the destination object.

\n

This new client-side checksum calculation is implemented in the latest version of the AWS SDKs. When you use an old SDK or custom code that doesn’t pre-compute checksums, Amazon S3 computes the checksum on all new objects it receives and stores it in the object’s metadata, even for multipart uploads.

\n

Pricing and availability
This extended checksum computation and storage is available in all AWS Regions at no additional cost.

\n

Update your AWS SDK and AWS CLI today to automatically benefit from this additional integrity protection for data in transit.

\n

To learn more about data integrity protection on Amazon S3, visit Checking object integrity in the Amazon S3 User Guide.

\n

-- seb

", "url": "https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/", "title": "Introducing default data integrity protections for new objects in Amazon S3", "date_modified": "2024-12-01T20:46:36.000Z" }, { "content_html": "

AWS Verified Access provides secure access to your corporate applications and resources without a virtual private network (VPN). We launched Verified Access in preview at re:Invent 2 years ago as a way to provide secure, VPN-less access to corporate applications, enabling organizations to manage network access based on identity and device security instead of IP addresses, which increases control and security over application access.

\n

Today, Verified Access is launching a preview of its secure, VPN-less access capabilities to non-HTTP(S) applications and resources, enabling zero trust access to corporate resources over protocols such as Secure Shell (SSH) and Remote Desktop Protocol (RDP).

\n

Organizations increasingly require secure, remote access to internal resources such as databases, remote desktops, and Amazon Elastic Compute Cloud (Amazon EC2) instances. Traditional VPN solutions, although effective for network access, often grant broad privileges and don’t support granular access controls, which can expose infrastructure with sensitive data. Although some organizations use bastion hosts to mediate access, this approach can create complexity and policy inconsistencies across HTTP(S) and non-HTTP(S) applications. With the rise of zero trust architectures, these gaps highlight the need for a secure access solution that extends consistent access policies across all applications and resources.

\n

Verified Access addresses these needs by providing zero trust access controls for your corporate applications and resources. By supporting protocols such as SSH, RDP, or Java Database Connectivity (JDBC) or Open Database Connectivity (ODBC), Verified Access simplifies your security operations. Now, you can establish uniform, context-aware access policies across your corporate applications and resources. Verified Access evaluates each access request in real time, making sure access is granted only to users who meet specific identity and device security requirements. Additionally, it eliminates the need for separate VPNs or bastion hosts, streamlining operations and reducing the risk of over-privileged access.

\n

One of my favorite capabilities is onboarding a group of resources by specifying their IP Classless Inter-Domain Routing (CIDR) and ports, rather than onboarding one resource at a time. Verified Access automatically creates DNS records for each active resource within the specified CIDR range. This eliminates the need for manual DNS configuration and users can therefore connect to new resources instantly.

\n

Using Verified Access for non-HTTPS access
Configuring Verified Access for non-HTTPS access isn’t very different from what exists today. You can read the blog post I wrote for the launch of the preview 2 years ago or the Get started with Verified Access tutorial to learn how to get started.

\n

Verified Access proposes two new types of endpoint targets: a target for one single resource and a target for multiple resources.

\n

With the network interface, load balancer, or RDS endpoint target you can provide access to an individual resource such as an Amazon Relational Database Service (Amazon RDS) instance or an arbitrary TCP application fronted by a Network Load Balancer or an elastic network interface. This type of target endpoint is defined by a combination of a target type (such as a load balancer or a network interface) and a range of TCP ports. Verified Access will provide a DNS name for each endpoint upon its creation. A Verified Access DNS name is assigned for each target. This is the name end users will use to securely access the resource.

\n

With network CIDR endpoint target, the resources are defined using an IP CIDR and port range. Through this type of endpoint target, you can easily provision secure access to ephemeral resources such as EC2 instances over protocols such as SSH and RDP. This is done without having to perform any actions such as creating or deleting endpoint targets each time a resource is added or removed. As long as these resources are assigned an IP address from the defined CIDR, Verified Access provides a unique public DNS record for each active IP detected in the defined CIDR.

\n

Here is a diagram of the setup for this demo.

\n

\"AWS

\n

Part 1: As a Verified Access administrator

\n

As a Verified Access administrator, I create the Verified Access instance, trust provider, access group, endpoint, and access policies, allowing access by the end user to the SSH server.

\n

For this demo, I configure a Verified Access network CIDR endpoint target. I select TCP as Protocol and Network CIDR as Endpoint type. I make sure the CIDR range is within the one of the VPC where my target resources are. I select the TCP Port ranges and the Subnets within the VPC.

\n

\"AVA

\n

This is a good moment to stretch your legs and refill your cup of coffee, it takes a few minutes to create the endpoint.

\n

Once, the status is \"✅\" Active, I launch an EC2 instance in a private Amazon Virtual Private Cloud (Amazon VPC). I enable SSH and configure the instance’s security group to only access requests coming from the VPC. A few minutes later, I can see the instance IP has been detected and assigned a DNS name to connect to from the Verified Access client application.

\n

I also have the option during the configuration to delegate my own DNS subdomain, such as secure.mycompany.com, and Verified Access will assign DNS names for the resources within that subdomain.

\n

\"AVA

\n

Create an access policy

\n

At this stage, there is no policy defined on the Verified Access endpoint. It will deny every request by default.

\n

On the Verified Access groups page, I select the Policy tab. Then I select the Modify Verified Access endpoint policy button to create an access policy.

\n

\"Verified

\n

I enter a policy allowing anybody who is authenticated and has an email address ending with @amazon.com. This is the email address I used for the user defined in AWS IAM Identity Center. Note that the name after context is the name I entered as Policy reference name when I created the Verified Access trust provider. The documentation page has the details of the policy syntax, the attributes, and the operators I can use.

\n
permit(principal, action, resource)\nwhen {\n    context.awsnewsblog.user.email.address like \"*@amazon.com\"\n};
\n

\"Verified

\n

After a few minutes, Verified Access updates the policy and becomes Active again.

\n

Distribute the configuration to clients

\n

The last task as a Verified Access administrator is to extract the JSON configuration file of the client applications.

\n

I retrieve the client application configuration file with the AWS Command Line Interface (AWS CLI). As a system administrator, I’ll distribute this configuration to each client machine.

\n
aws ec2 export-verified-access-instance-client-configuration \\\n     --verified-access-instance-id \"vai-0dbf2c4c011083069\"\n\n{\n    \"Version\": \"1.0\",\n    \"VerifiedAccessInstanceId\": \"vai-0dbf2c4c011083069\",\n    \"Region\": \"us-east-1\",\n    \"DeviceTrustProviders\": [],\n    \"UserTrustProvider\": {\n        \"Type\": \"iam-identity-center\",\n        \"Scopes\": \"verified_access:application:connect\",\n        \"Issuer\": \"https://identitycenter.amazonaws.com/ssoins-xxxx\",\n        \"PkceEnabled\": true\n    },\n    \"OpenVpnConfigurations\": [\n        {\n            \"Config\": \"Y2...bWU=\",\n            \"Routes\": [\n                {\n                    \"Cidr\": \"2600:1f10:4a02:8700::/57\"\n                }\n            ]\n        }\n    ]\n}
\n

Now that I have a resource to connect to and the Verified Access infrastructure in place, let me show you the end user experience to access a network endpoint.

\n

Part 2: As an end user

\n

As the end user, I receive a link to download and install the Verified Access Connectivity Client application. We support Windows and macOS clients at the time of this writing.

\n

I install the configuration file I received from my administrator. I use ClientConfig1.json as the file name and I copy the file to C:\\ProgramData\\Connectivity Client on Windows or /Library/Application\\ Support/Connectivity\\ Client on macOS.

\n

This is the same configuration file for all users, and the system administrator might push the file to all client machines using an endpoint management tool.

\n

I start the Connectivity Client application. I choose Sign in to start the authentication sequence.

\n

\"AVAThe authentication opens my web browser on the authentication page of my identity provider. The exact screen and login sequence varies from one provider to the other. After I’m authenticated, the Connectivity Client creates the secure tunnel to access my resource, an EC2 instance for this demo.

\n \n \n \n \n \n \n \n
\"AVA\"AVA
\n

Once the status is Connected, I can securely connect to the resource, using the DNS name provided by Verified Access. In a terminal application, I type the ssh command to start the connection.

\n

For this demo, I configured a delegated DNS domain secure.mycompany.com for Verified Access. The DNS address I received for the EC2 instance is 10-0-1-199.awsnews.secure.mycompany.com.

\n
$ ssh -i mykey.pem ec2-user@10-0-1-199.awsnews.secure.mycompany.com\n\n   ,     #_\n   ~\\_  ####_        Amazon Linux 2023\n  ~~  \\_#####\\\n  ~~     \\###|\n  ~~       \\#/ ___   https://aws.amazon.com/linux/amazon-linux-2023\n   ~~       V~' '->\n    ~~~         /\n      ~~._.   _/\n         _/ _/\n       _/m/'\nLast login: Sat Nov 17 20:17:46 2024 from 1.2.3.4\n\n$
\n

Availability and pricing
Verified Access is available as a public preview in 18 AWS Regions: US East (Ohio, N. Virginia), US West (N. California, Oregon), Asia Pacific (Jakarta, Mumbai, Seoul, Singapore, Sydney, Tokyo), Canada (Central), Europe (Frankfurt, Ireland, London, Milan, Stockholm), Israel (Tel Aviv), and South America (São Paulo).

\n

You’re charged for each hour that your non-HTTP(S) Verified Access endpoint remains active and per connection. The first 100 connections per month on each Verified Access endpoint are free. For more information, refer to AWS Verified Access Pricing.

\n

With Verified Access for HTTP(S) and non-HTTP(S) applications you can unify the access controls to your private applications and systems and apply zero trust policies uniformly to all applications, and SSH, RDP, and HTTP(S) resources. It reduces the complexity of your network infrastructure and helps you to implement zero-trust access to your applications and resources. Finally, it adapts to your growing infrastructure, automating DNS setup and supporting large-scale deployments without resource-specific registration.

\n

Go, try Verified Access today, and share your feedback with the team!

\n

-- seb

", "url": "https://aws.amazon.com/blogs/aws/aws-verified-access-now-supports-secure-access-to-resources-over-non-https-protocols/", "title": "AWS Verified Access now supports secure access to resources over non-HTTP(S) protocols (in preview)", "date_modified": "2024-12-01T20:40:48.000Z" }, { "content_html": "

Today, we’re announcing the general availability of Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode, a new capability to streamline Kubernetes cluster management for compute, storage, and networking, from provisioning to on-going maintenance with a single click. You can achieve higher agility, performance, and cost-efficiency by eliminating the operational overhead of managing the cluster infrastructure required to run production-grade Kubernetes applications at scale on Amazon Web Services (AWS).

\n

Customers choose Amazon EKS because they can use the open standards and portability of Kubernetes with the security, scalability, and availability of AWS cloud. While Kubernetes gives advanced customers deep controls over application operations, other customers find managing the components required for production-grade Kubernetes applications to be complex and labor-intensive.

\n

With the EKS Auto Mode, you can automate cluster management without deep Kubernetes expertise, because it selects optimal compute instances, dynamically scales resources, continuously optimizes costs, manages core add-ons, patches operating systems, and integrates with AWS security services. AWS expands its operational responsibility in EKS Auto Mode compared to customer-managed infrastructure in your EKS clusters. In addition to the EKS control plane, AWS will configure, manage, and secure the AWS infrastructure in EKS clusters that your applications need to run.

\n

You can now get started quickly, improve performance, and reduce overhead, enabling you to focus on building applications that drive innovation instead of on cluster management tasks. EKS Auto Mode also reduces the work required to acquire and run cost-efficient GPU-accelerated instances so that your generative AI workloads have the capacity they need when they need it.

\n

Get started with Amazon EKS Auto Mode
To get started, go to the Amazon EKS console and start to create your EKS cluster. You’ll have two options, Quick configuration (with EKS Auto Mode) and Custom configuration.

\n

\n

After you choose quick configuration, enter your cluster name and Kubernetes version, IAM roles, VPC subnets. You can view configuration default values in EKS Auto Mode whether you can edit after the cluster is created.

\n

\n

EKS Auto Mode enables the following Kubernetes capabilities in your EKS cluster:

\n \n

When you choose Create, your EKS cluster with Auto Mode will be deployed in minutes with a single click.

\n

If you choose the custom configuration option, you can customize other aspects of your cluster. You can use EKS Auto Mode in this option too.

\n

\n

You can also create an EKS Auto Mode cluster using AWS Command Line Interface (AWS CLI), eksctl, and AWS CloudFormation. Run the following eksctl command to create a new EKS Auto Mode cluster with:

\n
$ eksctl create cluster --name=<cluster-name> --enable-auto-mode
\n

To learn more, visit Create cluster with EKS Auto Mode in the Amazon EKS User Guide.

\n

If you want to enable EKS Auto Mode for an existing EKS cluster, choose Manage in the EKS Auto Mode section of the Overview tab in the EKS cluster detail page.

\n

\n

Select the box next to Use EKS Auto Mode to enable the EKS Auto Mode. You can unselect the EKS Auto Mode that will be configured in the cluster. The default is to create both a system and a default node pool and a node class.

\n

\n

You can also migrate from Karpenter, EKS Managed Node Groups, and EKS Fargate to EKS Auto Mode. To learn more, visit Enable EKS Auto Mode on existing EKS clusters in the Amazon EKS User Guide.

\n

To meet your workload requirements, you can configure specific aspects of your EKS Auto Mode clusters. While EKS Auto Mode manages most infrastructure components automatically, you can customize node networking settings, node compute resources, storage class settings, and application load balancing behaviors while maintaining the benefits of automated infrastructure management. To learn more, visit Change EKS Auto cluster settings in the Amazon EKS User Guide.

\n

Now, you can deploy different types of workloads to Amazon EKS clusters running in EKS Auto Mode. We provide key workload patterns including sample applications, load-balanced web applications, stateful workloads using persistent storage, and workloads with specific node placement requirements. Each example includes complete manifests and step-by-step deployment instructions that you can use as templates for your own applications. To learn more, visit Run workloads in EKS Auto Mode clusters in the Amazon EKS User Guide.

\n
\n

Now available
Amazon EKS Auto Mode is now available in all commercial AWS Regions except China Regions where Amazon EKS is available. You can enable EKS Auto Mode in any EKS cluster running Kubernetes 1.29 and above with no upfront fees or commitments—you pay for the management of the compute resources provisioned, in addition to your regular EC2 costs. To learn more, visit Amazon EKS pricing page.

\n

Please register for the online webinar: Simplifying Kubernetes operations with Amazon EKS Auto Mode on December 12, 2024 to learn more about how EKS Auto Mode can accelerate your time to deploy workloads to production and reduce the operational overheads of Kubernetes. To learn more, visit Automate cluster infrastructure with EKS Auto Mode in the Amazon EKS User Guide.

\n

Give EKS Auto Mode a try in the Amazon EKS console and send feedback to AWS re:Post for EKS or through your usual AWS Support contacts.

\n

— Channy

\n
", "url": "https://aws.amazon.com/blogs/aws/streamline-kubernetes-cluster-management-with-new-amazon-eks-auto-mode/", "title": "Streamline Kubernetes cluster management with new Amazon EKS Auto Mode", "date_modified": "2024-12-01T20:35:33.000Z" }, { "content_html": "Comments", "url": "https://avi.im/blag/2024/s3-log/", "title": "Building a distributed log using S3 (under 150 lines of Go)", "date_modified": "2024-12-01T16:10:46.000Z" }, { "content_html": "Comments", "url": "https://bilbof.com/posts/kubernetes-on-hetzner", "title": "Kubernetes on Hetzner: cutting my infra bill by 75%", "date_modified": "2024-12-01T15:43:40.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=42287526", "title": "Ask HN: What's the one feature you'd want in a GitHub productivity tool?", "date_modified": "2024-12-01T10:21:39.000Z" }, { "content_html": "Comments", "url": "https://pablosblog.dev/posts/3/", "title": "Unlocking the Power of DynamoDB: A Developer's Journey", "date_modified": "2024-12-01T01:05:39.000Z" }, { "content_html": "Comments", "url": "https://lucvandonkersgoed.com/2024/11/16/making-aws-news-stupid-fast-with-smart-caching/", "title": "Making AWS News stupid fast with smart caching", "date_modified": "2024-11-29T17:52:48.000Z" }, { "content_html": "

Comments

", "url": "https://surfingcomplexity.blog/2024/11/28/quick-takes-on-the-latest-cloudflare-public-incident-write-up/", "title": "Quick takes on the latest Cloudflare public incident write-up", "date_modified": "2024-11-29T01:05:55.000Z" }, { "content_html": "

Today, we are announcing support for Elastic Fabric Adapter (EFA) and NVIDIA GPUDirect Storage (GDS) on Amazon FSx for Lustre. EFA is a network interface for Amazon EC2 instances that makes it possible to run applications requiring high levels of inter-node communications at scale. GDS is a technology that creates a direct data path between local or remote storage and GPU memory. With these enhancements, Amazon FSx for Lustre with EFA/GDS support provides up to 12 times higher (up to 1200 Gbps) per-client throughput compared to the previous FSx for Lustre version.

\n

You can use FSx for Lustre to build and run the most performance demanding applications, such as deep learning training, drug discovery, financial modeling, and autonomous vehicle development. As datasets grow and new technologies emerge, you can adopt increasingly powerful GPU and HPC instances such as Amazon EC2 P5, Trn1, and Hpc7a. Until now, when accessing FSx for Lustre file systems, the use of traditional TCP networking limited throughput to 100 Gbps for individual client instances. This adoption is driving the need for FSx for Lustre file systems to provide the performance necessary to optimally utilize the increasing network bandwidth of these cutting-edge EC2 instances when accessing large datasets.

\n

With EFA and GDS support in FSx for Lustre, you can now achieve up to 1,200 Gbps throughput per client instance (twelve times more throughput than previously) when using P5 GPU instances and NVIDIA CUDA in your applications.

\n

With this new capability, you can fully utilize the network bandwidth of the most powerful compute instances and accelerate your machine learning (ML) and HPC workloads. EFA enhances performance by bypassing the operating system and using the AWS Scalable Reliable Datagram (SRD) protocol to optimize data transfer. GDS further improves performance by enabling direct data transfer between the file system and GPU memory, bypassing the CPU and eliminating redundant memory copies.

\n

Let’s see how this works in practice.

\n

Creating an Amazon FSx for Lustre file system with EFA enabled
To get started, in the Amazon FSx console, I choose Create file system and then Amazon FSx for Lustre.

\n

I enter a name for the file system. In the Deployment and storage type section, I select Persistent, SSD and the new with EFA enabled option. I select 1000 MB/s/TiB in the Throughput per unit of storage section. With these settings, I enter 4.8 TiB for Storage capacity, which is the minimum supported with these settings.

\n

\"Console

\n

For networking, I use the default virtual private cloud (VPC) and an EFA-enabled security group. I leave all other options to their default values.

\n

\"Console

\n

I review all the options and proceed to create the file system. After a few minutes, the file system is ready to be used.

\n

Mounting an Amazon FSx for Lustre file system with EFA enabled from an Amazon EC2 instance
In the Amazon EC2 console, I choose Launch instance, enter a name for the instance, and select the Ubuntu Amazon Machine Image (AMI). For Instance type, I select trn1.32xlarge.

\n

\"Console

\n

In Network settings, I edit the default settings and select the same subnet used by the FSx Lustre file system. In Firewall (security groups), I select three existing security groups: the EFA-enabled security group used by the FSx for Lustre file system, the default security group, and a security group that provides Secure Shell (SSH) access.

\n

\"Console

\n

In Advanced network configuration, I select ENA and EFA as Interface type. Without this setting, the instance would use traditional TCP networking and the connection with the FSx for Lustre file system would still be limited to 100 Gbps in throughput.

\n

\"Console

\n

To have more throughput, I can add more EFA network interfaces, depending on the instance type.

\n

I launch the instance and, when the instance is ready, I connect using EC2 Instance Connect and follow the instructions for installing the Lustre client in the FSx for Lustre User Guide and configuring EFA clients.

\n

Then, I follow the instructions for mounting an FSx for Lustre file system from an EC2 instance.

\n

I create a folder to use as mount point:

\n
\n 
sudo mkdir -p /fsx
\n
\n

I select the file system in the FSx console and lookup the DNS name and Mount name. Using these values, I mount the file system:

\n
\n 
sudo mount -t lustre -o relatime,flock file_system_dns_name@tcp:/mountname /fsx
\n
\n

EFA is automatically used when you access an EFA-enabled file system from client instances that support EFA and are using Lustre version 2.15 or higher.

\n

Things to know
EFA and GDS support is available today with no additional cost on new Amazon FSx for Lustre file systems in all AWS Regions where persistent 2 is offered. FSx for Lustre automatically uses EFA when customers access an EFA-enabled file system from client instances that support EFA, without requiring any additional configuration. For a list of EC2 client instances that support EFA, see supported instance types in the Amazon EC2 User Guide. This network specifications table describes network bandwidths and EFA support for instance types in the accelerated computing category.

\n

To use EFA-enabled instances with FSx for Lustre file systems, you must use Lustre 2.15 clients on Ubuntu 22.04 with kernel 6.8 or higher.

\n

Note that your client instances and your file systems must be located in the same subnet within your Amazon Virtual Private Cloud (Amazon VPC) connection.

\n

GDS is automatically supported on EFA-enabled file systems. To use GDS with your FSx for Lustre file systems, you need the NVIDIA Compute Unified Device Architecture (CUDA) package, the open source NVIDIA driver, and the NVIDIA GPUDirect Storage Driver installed on your client instance. These packages come preinstalled on the AWS Deep Learning AMI. You can then use your CUDA-enabled application to use GPUDirect storage for data transfer between your file system and GPUs.

\n

When planning your deployment, note that EFA-enabled file systems have larger minimum storage capacity increments than file systems that are not EFA-enabled. For instance, if you choose the 1,000 MB/s/TiB throughput tier, the minimum storage capacity for EFA-enabled file systems starts at 4.8 TiB as compared to 1.2TB for FSx for Lustre file systems not enabling EFA. If you’re looking to migrate your existing workloads, you can use AWS DataSync to move your data from an existing file system to a new one that supports EFA and GDS.

\n

For maximum flexibility, FSx for Lustre maintains compatibility with both EFA and non-EFA workloads. When accessing an EFA-enabled file system, traffic from non-EFA client instances automatically flows over traditional TCP/IP networking using Elastic Network Adapter (ENA), allowing seamless access for all workloads without any additional configuration.

\n

To learn more about EFA and GDS support on FSx for Lustre, including detailed setup instructions and best practices, visit the Amazon FSx for Lustre documentation. Get started today and experience the fastest storage performance available for your GPU instances in the cloud.

\n

— Danilo

\n

Update 11/27: post updated to reflect 12x throughput

", "url": "https://aws.amazon.com/blogs/aws/amazon-fsx-for-lustre-unlocks-full-network-bandwidth-and-gpu-performance/", "title": "Amazon FSx for Lustre increases throughput to GPU instances by up to 12x", "date_modified": "2024-11-27T22:13:37.000Z" }, { "content_html": "Comments", "url": "https://www.bbc.co.uk/news/articles/cje050wz22qo", "title": "London's 850-year-old food markets to close", "date_modified": "2024-11-27T21:36:38.000Z" }, { "content_html": "Comments", "url": "https://kerkour.com/aws-s3-vs-cloudflare-r2-price-performance-user-experience", "title": "Comparing AWS S3 with Cloudflare R2: Price, Performance and User Experience", "date_modified": "2024-11-27T15:26:52.000Z" }, { "content_html": "Comments", "url": "https://github.com/vordenken/AutoPiP", "title": "Show HN: AutoPiP – Safari extension for automatic Picture-in-Picture mode", "date_modified": "2024-11-27T10:36:00.000Z" }, { "content_html": "Comments", "url": "https://benhouston3d.com/blog/why-i-left-kubernetes-for-google-cloud-run", "title": "I Didn't Need Kubernetes, and You Probably Don't Either", "date_modified": "2024-11-27T02:26:19.000Z" }, { "content_html": "

You can now specify a desired completion duration (15 minutes to 48 hours) when you copy an Amazon Elastic Block Store (Amazon EBS) snapshot within or between AWS Regions and/or accounts. This will help you to meet time-based compliance and business requirements for critical workloads. For example:

\n

Testing – Distribute fresh data on a timely basis as part of your Test Data Management (TDM) plan.

\n

Development – Provide your developers with updated snapshot data on a regular and frequent basis.

\n

Disaster Recovery – Ensure that critical snapshots are copied in order to meet a Recovery Point Objective (RPO).

\n

Regardless of your use case, this new feature gives you consistent and predictable copies. This does not affect the performance or reliability of standard copies—you can choose the option and timing that works best for each situation.

\n

Creating a Time-Based Snapshot Copy
I can create time-based snapshot copies from the AWS Management Console, CLI (copy-snapshot), or API (CopySnapshot). While working on this post I created two EBS volumes (100 GiB and 1 TiB), filled each one with files, and created snapshots:

\n

\n

To create a time-based snapshot, I select the source as usual and choose Copy snapshot from the Action menu. I enter a description for the copy, choose the us-east-1 AWS Region as the destination, select Enable time-based copy, and (because this is a time-critical snapshot), enter a 15 minute Completion duration:

\n

\n

When I click Copy snapshot, the request will be accepted (and the copy will become Pending) only if my account’s throughput quotas are not already exceeded due to the throughput consumed by other active copies that I am making to the destination region. If the account level throughput quota is already exceeded, the console will display an error.

\n

I can click Launch copy duration calculator to get a better idea of the minimum achievable copy duration for the snapshot. I open the calculator, enter my account’s throughput limit, and choose an evaluation period:

\n

\n

The calculator then uses historical data collected over the course of previous snapshot copies to tell me the minimum achievable completion duration. In this example I copied 1,800,000 MiB in the last 24 hours; with time-based copy and my current account throughput quota of 2000 MiB/second I can copy this much data in 15 minutes.

\n

\n

While the copy is in progress, I can monitor progress using the console or by calling DescribeSnapshots and examining the progress field of the result. I can also use the following Amazon EventBridge events to take actions (if the copy operation crosses regions, the event is sent in the destination region):

\n

copySnapshot – Sent after the copy operation completes.

\n

copyMissedCompletionDuration – Sent if the copy is still pending when the deadline has passed.

\n

Things to Know
And that’s just about all there is to it! Here’s what you need to know about time-based snapshot copies:

\n

CloudWatch Metrics – The SnapshotCopyBytesTransferred metric is emitted in the destination region, and reflect the amount of data transferred between the source and destination region in bytes.

\n

Duration – The duration can range from 15 minutes to 48 hours in 15 minute increments, and is specified on a per-copy basis.

\n

Concurrency – If a snapshot is being copied and I initiate a second copy of the same snapshot to the same destination, the duration for the second one starts when the first one is completed.

\n

Throughput – There is a default per-account limit of 2000 MiB/second between each source and destination pair. If you need additional throughput in order to meet your RPO you can request an increase via the AWS Support Center. Maximum per-snapshot throughput is 500 MiB/second and cannot be increased.

\n

Pricing – Refer to the Amazon EBS Pricing page for complete pricing information.

\n

Regions – Time-based snapshot copies are available in all AWS Regions.

\n

— Jeff;

", "url": "https://aws.amazon.com/blogs/aws/time-based-snapshot-copy-for-amazon-ebs/", "title": "Time-based snapshot copy for Amazon EBS", "date_modified": "2024-11-26T22:31:36.000Z" }, { "content_html": "Comments", "url": "https://www.warp.dev/blog/lifting-login-requirement", "title": "Warp terminal – no more login required", "date_modified": "2024-11-26T17:14:50.000Z" }, { "content_html": "Comments", "url": "https://status.flyio.net", "title": "Fly.io outage – resolved", "date_modified": "2024-11-26T01:47:25.000Z" }, { "content_html": "Comments", "url": "https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/", "title": "Amazon S3 Adds Put-If-Match (Compare-and-Swap)", "date_modified": "2024-11-25T22:11:31.000Z" }, { "content_html": "
\n
\n
\n Ben Schwartz voices the titular character in Sonic the Hedgehog 3.\n\n
\n
\n
\n

Some lucky folks got a heads-up last week that a trailer for Sonic the Hedgehog 3 was about to drop when Paramount Pictures released a playable Sega Genesis gaming cartridge to select individuals. Hidden among a mini-game and character posters and accessible by entering a cheat code was the trailer release date. True to its word, Paramount just dropped the final trailer for the third film in the successful franchise. All our favorite characters are back, as well as a couple of new ones, most notably a new villain familiar to fans of the games: Shadow, voiced by Keanu Reeves.

\n

(Spoilers for the first two films below.)

\n

As previously reported, in the first film, Sonic (Ben Schwartz) teamed up with local town sheriff Tom Wachowski (James Marsden) to stop the sinister mad scientist Dr. Robotnik (Jim Carrey). Robotnik wanted to catch and experiment on the hedgehog, and if he could also frame Tom as a domestic terrorist, even better.

Read full article

\n

Comments

", "url": "https://arstechnica.com/culture/2024/11/keanu-reeves-voices-arch-villain-shadow-in-sonic-3-trailer/", "title": "Keanu Reeves voices archvillain Shadow in Sonic 3 trailer", "date_modified": "2024-11-25T17:36:35.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/model-context-protocol", "title": "Model Context Protocol", "date_modified": "2024-11-25T16:14:22.000Z" }, { "content_html": "

Comments

", "url": "https://medium.com/@jgrodman/clickhouse-fast-deduplicated-reads-de2026a4e0ad", "title": "ClickHouse — fast, deduplicated reads", "date_modified": "2024-11-25T12:04:16.000Z" }, { "content_html": "Comments", "url": "https://github.com/github/roadmap/discussions/1014", "title": "GitHub removes 42 \"basic functionality\" features from their roadmap", "date_modified": "2024-11-25T08:48:26.000Z" }, { "content_html": "Comments", "url": "https://github.com/github/roadmap/discussions/1014", "title": "Deprecating outdated issues on the GitHub public roadmap", "date_modified": "2024-11-25T08:48:26.000Z" }, { "content_html": "

Comments

", "url": "https://avi.im/blag/2024/zero-disk-architecture/", "title": "Zero Disk Architecture", "date_modified": "2024-11-24T15:07:19.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/infra-team-stays-small", "title": "How Tailscale's infra team stays small", "date_modified": "2024-11-23T03:28:46.000Z" }, { "content_html": "

Today, I’m excited to introduce a new and improved version of AWS Systems Manager that brings a highly requested cross-account, and cross-Region experience for managing nodes at scale.

\n

The new System Manager experience provides centralized visibility of all your managed nodes which include various infrastructure types, such as Amazon Elastic Compute Cloud (EC2) instances, containers, virtual machines on other cloud providers, on-premise servers, and edge Internet of Things (IoT) devices. They are referred to as “managed nodes” when they have the Systems Manager Agent (SSM Agent) installed and are connected to Systems Manager.

\n

If an SSM Agent stops working on a node for whatever reason, then Systems Manager loses connection to it and that node is then referred to as an “unmanaged node.” With the new update, Systems Manager can also help you to easily discover and troubleshoot unmanaged nodes. You can run and even schedule an automated diagnosis that provides you with recommended runbooks that you can execute to fix any issues and reestablish connection so they become managed nodes again.

\n

Systems Manager is also now integrated with Amazon Q Developer, the most capable generative AI–powered assistant for software development. You can ask questions about your managed nodes to Amazon Q Developer using natural language and it will provide you with rapid insights plus links straight to Systems Manager where you can perform actions or continue to explore further.

\n

With this release, you can also use AWS Organizations, to allow a delegated administrator to centrally manage nodes across the organization thanks to the new integration with Systems Manager.

\n

\"the

\n

Let’s examine a quick example that helps to demonstrate some of these new capabilities.

\n

Imagine a scenario where you are a cloud platform engineer leading a migration plan aiming to replace all nodes running Windows Server 2016 Datacenter in the organization. Let’s use the new Systems Manager experience to quickly gather information about all the nodes that needs to be included in our plan.

\n

Step 1 – Asking Amazon Q Developer
The easiest starting point is using Amazon Q Developer to ask what you want to find using natural language. Using the AWS Console, I open the Amazon Q chatbot and type Find all of my managed nodes running Microsoft Windows Server 2016 Datacenter in my organization.

\n

Amazon Q quickly comes back with an answer: it tells us that there are ten nodes that fit the criteria and provides a list with an overview of each one.

\n

\n

There is also a link that redirects to the new Explore nodes page in System Manager where we can learn more information. Let’s follow it.

\n

Step 2 – Reviewing our infrastructure
The Explore nodes page provides a comprehensive overview of all managed nodes across your organization, with options to group and filter results for quick access. In this case, we can see that the results are already filtered by Operating system name providing us with a list of all the nodes that are running Microsoft Windows Server 2016 Datacenter.

\n

\n

This is a great start! We could just finish here by downloading the report and add those nodes to our migration plan, however, this page only shows you information about your managed nodes. Could it be that there are unmanaged nodes that need to included in our plan? Let’s find out.

\n

Step 3 – Handling unmanaged nodes
Open the menu, and navigate to the Review node insights page. Here you can see a dashboard with widgets that provide insightful interactive charts that you can use to drill down and discover more information about your nodes or even take actions. For example, the Managed node types pie chart shows the types of managed nodes we have whereas the SSM Agent versions graph provides us with an overview of all the different versions of SSM Agent running on them. You can also customize this view by adding and replacing widgets.

\n

We want to investigate any unmanaged nodes to make sure we don’t miss any that may need to be added to our migration plan. The Node summary widget clearly shows that there are two unmanaged nodes. This could mean that these nodes don’t have the SSM Agent installed in which case we will need to investigate them manually. However, it could also just mean there are issues with the SSM agent permissions or network connectivity preventing Systems Manager from managing these nodes and treating them like any other managed node. The new Systems Manager experience allows you easily troubleshoot and remediate SSM Agents issues so let’s attempt to do this now.

\n

Start by selecting the piece of the chart displaying our unmanaged nodes. This pops up an option to initiate a comprehensive diagnosis of all our unmanaged nodes with only one click. Let’s run this.

\n

\n

The diagnosis reviews key configurations such as missing virtual private cloud (VPC) endpoints, misconfigured VPC DNS settings, and misconfigured instance security groups that may be preventing the SSM Agent from connecting to Systems Manager. After the scanning is complete, we can see that it displays two Misconfigured VPC endpoint findings. It also gives you a link that you can use to open a side panel containing a recommended runbook that you can execute to solve the issues as well as links to relevant documentation.

\n

\n

Choosing to execute the recommended runbook presents you with a detailed preview of the changes which include a thorough overview of the actions it’s going to take in addition to the input parameters used, a link to view a breakdown of the steps involved, and the target nodes for this execution.

\n

\n

Let’s choose to go ahead and select Execute. Keep in mind that this may incur costs, so make sure to review them before executing. You can keep an eye on progress on this page as it goes through the steps to attempt to fix the issues on each node.

\n

\n

Aha! After the remediation is complete, we can see that Systems Manager has found and corrected issues with the SSM Agent with two nodes. This means that Systems Manager is able to connect with the SSM Agent running in those nodes successfully making them “managed nodes.” We can verify this by returning to the Explore nodes page and noticing that the count of “unmanaged nodes” has been reduced to zero now.

\n

Now that all of our nodes are managed, we’re ready to get a full list of all of those that need to be added to our migration plan.

\n

Step 4 – Downloading a report
Back on the Explore nodes page we can see that the count for nodes running Microsoft Windows Server 2016 Datacenter has gone up from ten to twelve! That means that those previously unmanaged nodes that we fixed through the automated diagnosis are indeed running our target operating system.

\n

\n

This is exactly what we need so we choose to download a Report. You give it a file name, and then choose from a few options such as which columns to include. In this case, we choose to download a CSV file with a row containing the column names.

\n

\n

That’s it! We have our CSV with detailed information about the nodes that need upgrading across our entire infrastructure. And the best part? You can also use Systems Manager to automate the upgrade once you’re ready to go ahead with the migration.

\n

Conclusion
Systems Manager is a critical tool for gaining visibility and control over your compute infrastructure and performing operational actions at scale. The new experience offers a centralized cross-account, cross-Region view of all your nodes in your AWS accounts, on-premises, and multicloud environments through a centralized dashboard, offering integration with Amazon Q Developer for natural language queries, and one-click SSM Agent troubleshooting. You can enable the new experience at no extra cost by navigating to the Systems Manager console and following the straightforward instructions.

\n

To learn more, see the documentation for more detail about the new Systems Manager experience.

\n

Check out this interactive demo for a full visual tour of this experience.

", "url": "https://aws.amazon.com/blogs/aws/introducing-a-new-experience-for-aws-system-manager/", "title": "Introducing a new experience for AWS Systems Manager", "date_modified": "2024-11-22T22:36:19.000Z" }, { "content_html": "Comments", "url": "https://beeb.li/blog/aws-lambda-rust-docker", "title": "Rust for AWS Lambda, the Docker Way", "date_modified": "2024-11-22T16:16:39.000Z" }, { "content_html": "Comments", "url": "https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-express-one-zone-append-data-object/", "title": "Amazon S3 now supports the ability to append data to an object", "date_modified": "2024-11-22T04:46:39.000Z" }, { "content_html": "

In November 2023, we announced Amazon CloudWatch Application Signals, an AWS built-in application performance monitoring (APM) solution, to solve the complexity associated with monitoring performance of distributed systems for applications hosted on Amazon EKS, Amazon ECS, and Amazon EC2. Application Signals automatically correlates telemetry across metrics, traces, and logs, to speed up troubleshooting and reduce application disruption. By providing an integrated experience for analyzing performance in the context of your applications, Application Signals gives you improved productivity focusing on the applications that support your most critical business functions.

\n

Today we’re announcing the availability of Application Signals for AWS Lambda to eliminate the complexities of manual setup and performance issues required to assess application health for Lambda functions. With CloudWatch Application Signals for Lambda, you can now collect application golden metrics (the incoming and outgoing volume of requests, latency, faults, and errors).

\n

AWS Lambda abstracts away the complexity of the underlying infrastructure, enabling you to focus on building your application without having to monitor server health. This allows you to shift your focus toward monitoring the performance and health of your applications, which is necessary to operate your applications at peak performance and availability. This requires deep visibility into performance insights such as volume of transactions, latency spikes, availability drops, and errors for your critical business operations and application programming interfaces (APIs).

\n

Previously, you had to spend significant time correlating disjointed logs, metrics, and traces across multiple tools to establish the root cause of anomalies, increasing mean time to recovery (MTTR) and operational costs. Additionally, building your own APM solutions with custom code or manual instrumentation using open source (OSS) libraries was time-consuming, complex, operationally expensive, and often resulted in increased cold start times and deployment challenges when managing large fleets of Lambda functions. Now, you can use Application Signals to seamlessly monitor and troubleshoot health and performance issues in serverless applications, without requiring any manual instrumentation or code changes from your application developers.

\n

How it works
Using the pre-built, standardized dashboards of Application Signals, you can identify the root cause of performance anomalies in just a few clicks by drilling down into performance metrics for critical business operations and APIs. This helps you visualize application topology which shows interactions between the function and its dependencies. In addition, you can define Service Level Objectives (SLOs) on your applications to monitor specific operations that matter most to you. An example of an SLO could be to set a goal that a webpage should render within 2000 ms 99.9 percent of the time in a rolling 28-day interval.

\n

Application Signals auto-instruments your Lambda function using enhanced AWS Distro for OpenTelemetry (ADOT) libraries. This delivers better performance such as lower cold start latency,
memory consumption, and function invocation duration, so you can quickly monitor your applications.

\n

I have an existing Lambda function appsignals1 and I will configure Application Signals in the Lambda Console to collect various telemetry on this application.

\n

In the Configuration tab of the function I select Monitoring and operations tools to enable both the Application signals and the Lambda service traces.

\n

\n

\n

\n

I have an application myAppSignalsApp that has this Lambda function attached as a resource. I’ve defined an SLO for my application to monitor specific operations that matter most to me. I’ve defined a goal that states that the application executes within 10 ms 99.9 percent of the time in a rolling 1-day interval.

\n

\n

It can take 5-10 minutes for Application Signals to discover the function after it’s been invoked. As a result you’ll need to refresh the Services page before you can see the service.

\n

\n

Now I’m in the Services page and I can see a list of all my Lambda functions that have been discovered by Application Signals. Any telemetry that is emitted will be displayed here.

\n

\n

I can then visualize the complete application topology from the Service Map and quickly spot anomalies across my service’s individual operations and dependencies, using the newly collected metrics of volume of requests, latency, faults, and errors. To troubleshoot, I can click into any point in time for any application metric graph to discover correlated traces and logs related to that metric, to quickly identify if issues impacting end users are isolated to an individual task or deployment.

\n

\n

Available now
Amazon CloudWatch Application Signals for Lambda is now generally available and you can start using it today in all AWS Regions where Lambda and Application Signals are available. Today, Application Signals is available for Lambda functions that use Python and Node.js managed runtimes. We’ll continue to add support for other Lambda runtimes in near future.

\n

To learn more, visit the AWS Lambda developer guide and Application Signals developer guide. You can submit your questions to AWS re:Post for Amazon CloudWatch, or through your usual AWS Support contacts.

\n

– Veliswa.

", "url": "https://aws.amazon.com/blogs/aws/track-performance-of-serverless-applications-built-using-aws-lambda-with-application-signals/", "title": "Track performance of serverless applications built using AWS Lambda with Application Signals", "date_modified": "2024-11-21T20:50:46.000Z" }, { "content_html": "Comments", "url": "https://github.com/janderland/fql", "title": "FQL: A KV Query Language", "date_modified": "2024-11-20T15:50:39.000Z" }, { "content_html": "Comments", "url": "https://underjord.io/how-i-use-erlang-hot-code-updates.html", "title": "Using Erlang hot code updates", "date_modified": "2024-11-19T20:29:03.000Z" }, { "content_html": "
\"Source:

Source: Transit

\n

Earlier this month, Transit, one of my favorite apps of all time, gained an impressive new feature: the app is now able to track your train and warn you when you are about to reach your destination even when your train is underground. Previously, Transit had to rely on GPS and cellular service to precisely locate your train on its route, which meant it couldn’t reliably function as soon as you entered a subway tunnel.

\n

The way they have been able to achieve this is fascinating. Transit now utilizes the iPhone’s built-in accelerometer and analyzes its patterns to identify when the vehicle you boarded is in motion, and every time it reaches a station. The company’s account of the whole process is nothing short of impressive. The team spent a week riding buses and trains to collect data and proceeded to create an entirely new prediction model that is able to count down the underground stations that you will need to ride through to reach your destination. Transit says the model works completely offline and on-device.

\n

I know I’m going to give this new feature a try as soon as I get a chance to ride the Paris Métro next week.

\n

→ Source: blog.transitapp.com

", "url": "https://www.macstories.net/linked/transit-can-now-track-underground-trains-without-gps/", "title": "Transit Can Now Track Underground Trains without GPS", "date_modified": "2024-11-18T13:31:04.000Z" }, { "content_html": "Comments", "url": "https://blog.mbrt.dev/posts/transactional-object-storage/", "title": "Transactional Object Storage?", "date_modified": "2024-11-17T13:20:33.000Z" }, { "content_html": "Comments", "url": "https://github.com/hengyoush/kyanos", "title": "Kyanos: eBPF-based network issue analysis tool", "date_modified": "2024-11-16T05:34:57.000Z" }, { "content_html": "

AWS Identity and Access Management (IAM) is launching a new capability allowing security teams to centrally manage root access for member accounts in AWS Organizations. You can now easily manage root credentials and perform highly privileged actions.

\n

Managing root user credentials at scale
For a long time, Amazon Web Services (AWS) accounts were provisioned with highly privileged root user credentials, which had unrestricted access to the account. This root access, while powerful, also posed significant security risks. Each AWS account’s root user had to be secured by adding layers of protection like multi-factor authentication (MFA). Security teams were required to manage and secure these root credentials manually. The process involved rotating credentials periodically, storing them securely, and making sure that the credentials complied with security policies.

\n

As our customers expanded their AWS environments, this manual approach became cumbersome and prone to error. For example, large enterprises operating hundreds or thousands of member accounts struggled to secure root access consistently across all accounts. The manual intervention not only added operational overhead but also created a lag in account provisioning, preventing full automation and increasing security risks. Root access, if not properly secured, could lead to account takeovers and unauthorized access to sensitive resources.

\n

Furthermore, whenever specific root actions such as unlocking an Amazon Simple Storage Service (Amazon S3) bucket policy or an Amazon Simple Queue Service (Amazon SQS) resource policy were required, security teams had to retrieve and use root credentials, which only increased the attack surface. Even with rigorous monitoring and strong security policies, maintaining long-term root credentials opened doors to potential mismanagement, compliance risks, and manual errors.

\n

Security teams began seeking a more automated, scalable solution. They needed a way to not only centralize the management of root credentials but also programmatically manage root access without needing long-term credentials in the first place.

\n

Centrally manage root access
With the new ability to centrally manage root access, we address the longstanding challenge of managing root credentials across multiple accounts. This new capability introduces two essential capabilities: the central management of root credentials and root sessions. Together, they offer security teams a secure, scalable, and compliant way to manage root access across AWS Organizations member accounts.

\n

Let’s first discuss the central management of root credentials. With this capability, you can now centrally manage and secure privileged root credentials across all accounts in AWS Organizations. Root credentials management allows you to:

\n \n

But how can we make sure it remains possible to perform selected root actions on the accounts? This is the second capability we launch today: root sessions. It offers a secure alternative to maintaining long-term root access. Instead of manually accessing root credentials whenever privileged actions are required, security teams can now gain short-term, task-scoped root access to member accounts. This capability makes sure that actions such as unlocking S3 bucket policies or SQS queue policies can be performed securely without the need for long-term root credentials.

\n

Root sessions key benefits include:

\n \n

This new capability does not grant full root access. It provides temporary credentials for performing one of these five specific actions. The first three actions are possible with central management of root user credentials. The last two come when enabling root sessions.

\n \n

How to obtain root credentials on a member account
In this demo, I show you how to prepare your management account, create a member account without root credentials, and obtain temporary root credentials to make one of the five authorized API call on the member account. I assume you have an organization already created.

\n

First, I create a member account.

\n
aws organizations create-account    \\\n     --email stormacq+rootaccountdemo@amazon.com \\\n     --account-name 'Root User Demo account'\n{\n    \"CreateAccountStatus\": {\n        \"Id\": \"car-695abd4ee1ca4b85a34e5dcdcd1b944f\",\n        \"AccountName\": \"Root User Demo account\",\n        \"State\": \"IN_PROGRESS\",\n        \"RequestedTimestamp\": \"2024-09-04T20:04:09.960000+00:00\"\n    }\n}
\n

Then, I enable the two new capabilities on my management account. Don’t worry, these commands don’t alter the behavior of the accounts in any way other than enabling use of the new capability.

\n
➜  aws organizations enable-aws-service-access \\\n        --service-principal iam.amazonaws.com\n\n➜  aws iam enable-organizations-root-credentials-management\n{\n    \"OrganizationId\": \"o-rlrup7z3ao\",\n    \"EnabledFeatures\": [\n        \"RootCredentialsManagement\"\n    ]\n}\n\n➜  aws iam enable-organizations-root-sessions\n{\n    \"OrganizationId\": \"o-rlrup7z3ao\",\n    \"EnabledFeatures\": [\n        \"RootSessions\",\n        \"RootCredentialsManagement\"\n    ]\n}
\n

Alternatively, I can also use the console on the management account. On the IAM page, under Access management, I select Account settings.

\n

\"Root

\n

Now, I’m ready to make requests to obtain temporary root credentials. I have to pass one of the five managed IAM policies to scope down the credentials to a specific action.

\n
➜  aws sts assume-root    \\\n       --region us-east-1 \\\n       --target-principal <my member account id> \\\n       --task-policy-arn arn=arn:aws:iam::aws:policy/root-task/S3UnlockBucketPolicy \n\n{\n    \"Credentials\": {\n        \"AccessKeyId\": \"AS....XIG\",\n        \"SecretAccessKey\": \"ao...QxG\",\n        \"SessionToken\": \"IQ...SS\",\n        \"Expiration\": \"2024-09-23T17:44:50+00:00\"\n    }\n}
\n

Once I obtain the access key ID, the secret access key, and the session token, I use them as usual with the AWS Command Line Interface (AWS CLI) or an AWS SDKs.

\n

For example, I can pass these three values as environment variables.

\n
$ export AWS_ACCESS_KEY_ID=ASIA356SJWJITG32xxx\n$ export AWS_SECRET_ACCESS_KEY=JFZzOAWWLocoq2of5Exxx\n$ export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEMb//////////wEaCXVxxxx\n
\n

Now that I received the temporary credentials, I can make a restricted API call as root on the member account. First, I verify I now have root credentials. The Arn field confirms I’m working with the root user.

\n
\n# Call get Caller Identity and observe I'm root in the member account\n$ aws sts get-caller-identity\n{\n   \"UserId\": \"012345678901\",\n   \"Account\": \"012345678901\",\n   \"Arn\": \"arn:aws:iam::012345678901:root\"\n}\n
\n

Then, I use the delete-bucket-policy from S3 to remove an incorrect policy that has been applied to a bucket. The invalid policy removed all bucket access for everybody. Removing such policy requires root credentials.

\n

aws s3api delete-bucket-policy --bucket my_bucket_with_incorrect_policy

\n

When there is no output, it means the operation is successful. I can now apply a correct access policy to this bucket.

\n

Credentials are valid only for 15 minutes. I wrote a short shell script to automate the process of getting the credentials as JSON, exporting the correct environment variables, and issuing the command I want to run as root.

\n

Availability
Central management of root access is available at no additional cost in all AWS Regions except AWS GovCloud (US) and AWS China Regions, where there is no root user. Root sessions are available everywhere.

\n

You can start using it through the IAM console, AWS CLI or AWS SDK. For more information, visit AWS account root user in our documentation and follow best practices for securing your AWS accounts.

\n

-- seb

", "url": "https://aws.amazon.com/blogs/aws/centrally-managing-root-access-for-customers-using-aws-organizations/", "title": "Centrally managing root access for customers using AWS Organizations", "date_modified": "2024-11-15T16:56:44.000Z" }, { "content_html": "Comments", "url": "https://www.allthingsdistributed.com/2024/11/aws-lambda-turns-10-a-rare-look-at-the-doc-that-started-it.html", "title": "AWS Lambda PR/FAQ After 10 Years", "date_modified": "2024-11-15T16:16:05.000Z" }, { "content_html": "Comments", "url": "https://cmtops.dev/posts/building-observability-with-clickhouse/", "title": "Building Observability with ClickHouse", "date_modified": "2024-11-15T13:35:03.000Z" }, { "content_html": "Comments", "url": "https://www.redhat.com/en/blog/red-hat-contribute-comprehensive-container-tools-collection-cloud-native-computing-foundation", "title": "Red Hat to contribute container tech (Podman, bootc, ComposeFS...) to CNCF", "date_modified": "2024-11-14T17:59:27.000Z" }, { "content_html": "Comments", "url": "https://netflixtechblog.com/netflixs-distributed-counter-abstraction-8d0c45eb66b2", "title": "Netflix's Distributed Counter Abstraction", "date_modified": "2024-11-13T19:31:11.000Z" }, { "content_html": "
\"Source:

Source: Lux Camera.

\n

Lux Camera’s video camera app Kino has been updated to version 1.2, bringing a variety of new features and a redesigned icon. I covered the debut of Kino back in May and have been using it a lot lately because its design makes taking great-looking video so easy.

\n

At the heart of Kino’s 1.2 update is support for the latest iPhones. Kino now works with the iPhone 16 and 16 Pro’s Camera Control for making adjustments that previously were only possible by touching your iPhone’s screen.

\n
\"Kino

Kino Instant Grades. Source: Lux Camera.

\n

On the 16 Pro, the app also supports 4K video at 120fps with its Instant Grade feature enabled. That’s the feature that lets you pick a color grading preset created by video experts, including Stu Maschwitz, Sandwich Video, Evan Schneider, Tyler Stalman, and Kevin Ong. Version 1.2 of Kino lets you reorder those grades in its settings to make your favorites easier to access. Finally, Kino has added support for the following languages: Chinese, Dutch, French, German, Italian, Japanese, Korean, Portuguese, Russian, and Spanish.

\n

If you haven’t tried Kino before, it’s available on the App Store for $9.99.

\n

Access Extra Content and Perks

Founded in 2015, Club MacStories has delivered exclusive content every week for nearly a decade.

\n

What started with weekly and monthly email newsletters has blossomed into a family of memberships designed every MacStories fan.

\n

Club MacStories: Weekly and monthly newsletters via email and the web that are brimming with apps, tips, automation workflows, longform writing, early access to the MacStories Unwind podcast, periodic giveaways, and more;

\n

Club MacStories+: Everything that Club MacStories offers, plus an active Discord community, advanced search and custom RSS features for exploring the Club’s entire back catalog, bonus columns, and dozens of app discounts;

\n

Club Premier: All of the above and AppStories+, an extended version of our flagship podcast that’s delivered early, ad-free, and in high-bitrate audio.

\n

Learn more here and from our Club FAQs.

\n

Join Now", "url": "https://www.macstories.net/news/kino-1-2-adds-camera-control-support-and-higher-resolution-and-frame-rate-recording/", "title": "Kino 1.2 Adds Camera Control Support and Higher Resolution and Frame Rate Recording", "date_modified": "2024-11-13T17:22:42.000Z" }, { "content_html": "Comments", "url": "https://sst.dev/blog/container-support/", "title": "SST: Container Support", "date_modified": "2024-11-11T09:51:32.000Z" }, { "content_html": "Comments", "url": "https://mergiraf.org/", "title": "Mergiraf: a syntax-aware merge driver for Git", "date_modified": "2024-11-09T11:06:10.000Z" }, { "content_html": "Comments", "url": "https://jhftss.github.io/A-New-Era-of-macOS-Sandbox-Escapes/", "title": "Multiple new macOS sandbox escape vulnerabilities", "date_modified": "2024-11-08T06:10:14.000Z" }, { "content_html": "Comments", "url": "https://julien.danjou.info/p/theres-almost-no-gitlab", "title": "There's Almost No Gitlab", "date_modified": "2024-11-08T06:04:00.000Z" }, { "content_html": "Comments", "url": "https://bunny.net/blog/introducing-bunny-edge-scripting-a-better-way-to-build-and-deploy-applications-at-the-edge/", "title": "Edge Scripting: Build and run applications at the edge", "date_modified": "2024-11-07T18:17:20.000Z" }, { "content_html": "Comments", "url": "https://developer.apple.com/documentation/virtualization/accelerating_the_performance_of_rosetta", "title": "Accelerating the Performance of Rosetta in Linux VMs on Apple Silicon", "date_modified": "2024-11-07T08:02:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/dstackai/dstack", "title": "Dstack: An alternative to K8 for AI/ML tasks", "date_modified": "2024-11-05T16:56:31.000Z" }, { "content_html": "Comments", "url": "https://www.gitpod.io/blog/we-are-leaving-kubernetes", "title": "We're Leaving Kubernetes", "date_modified": "2024-11-04T14:41:28.000Z" }, { "content_html": "Comments", "url": "https://slawlor.github.io/ractor/quickstart/", "title": "Ractor – a Rust Actor Framework", "date_modified": "2024-11-03T01:47:23.000Z" }, { "content_html": "Comments", "url": "https://blog.howardjohn.info/posts/go-build-times/", "title": "Analyzing Go Build Times (2023)", "date_modified": "2024-11-01T14:22:53.000Z" }, { "content_html": "Comments", "url": "https://holos.run/docs/guides/helm/", "title": "Show HN: Holos – Configure Helm and Kustomize Holistically with Cue", "date_modified": "2024-10-29T12:58:16.000Z" }, { "content_html": "Comments", "url": "https://linderud.dev/blog/nixos-is-not-reproducible/", "title": "NixOS Is Not Reproducible", "date_modified": "2024-10-26T07:19:07.000Z" }, { "content_html": "

It took a crew of more than 100 talented modders and another hundred voice actors nearly five years to make Fallout: London. Just as they planned to release it, Bethesda came out with a \"next-gen upgrade\" of the mod's base game, Fallout 4, forcing the team to scramble and ultimately find a way to downgrade the game. When they finally released London, they then dealt with game-stopping bugs and quality issues that their small QA team could not have caught. It's been a long, maybe even post-apocalyptic road for these modders.

\n

A few updates later, Fallout: London is in much better shape. I've been able to put about 12 hours into it, and that, in itself, is essentially my review: it is worth that kind of time and more. If you can still enjoy Fallout 4, of course.

\n

Any Fallout fan waits a long time between official releases, so it can be tempting to go easy on any new offering, however spit-and-bailing-wire it may seem. But Fallout: London is a game in its own right, with a distinct look, vision, and stories to tell. You can find evidence of its unofficial mod-ness if you look around, but you're better off doing the Fallout thing: wandering, wondering, fighting, and occasionally talking to some messed-up weirdo.

Read full article

\n

Comments

", "url": "https://arstechnica.com/gaming/2024/10/fallout-london-is-a-huge-fallout-4-mod-that-is-now-playable-and-worth-playing/", "title": "Fallout: London is a huge Fallout 4 mod that is now playable—and worth playing", "date_modified": "2024-10-25T11:30:05.000Z" }, { "content_html": "Comments", "url": "https://github.com/open-feature", "title": "OpenFeature – a vendor-agnostic, community-driven API for feature flagging", "date_modified": "2024-10-25T01:26:45.000Z" }, { "content_html": "Comments", "url": "https://rsbuild.dev/", "title": "Rsbuild – A Better Vite?", "date_modified": "2024-10-25T01:12:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/resola-ai/rust-aws-tui", "title": "Show HN: Rust based AWS Lambda Logs Viewer (TUI)", "date_modified": "2024-10-25T00:53:59.000Z" }, { "content_html": "Comments", "url": "https://github.com/E-xyza/zigler", "title": "Zigler: Zig NIFs in Elixir", "date_modified": "2024-10-24T17:53:26.000Z" }, { "content_html": "Comments", "url": "https://danielquinn.org/blog/developing-with-docker/", "title": "Developing with Docker", "date_modified": "2024-10-24T14:17:23.000Z" }, { "content_html": "

Workflows, Cloudflare’s durable execution engine that allows you to build reliable, repeatable multi-step applications that scale for you, is now in open beta. Any developer with a free or paid Workers plan can build and deploy a Workflow right now: no waitlist, no sign-up form, no fake line around-the-block.

If you learn by doing, you can create your first Workflow via a single command (or visit the docs for the full guide):

\n 
npm create cloudflare@latest workflows-starter -- \\\n  --template \"cloudflare/workflows-starter\"
\n

Open the src/index.ts file, poke around, start extending it, and deploy it with a quick wrangler deploy.

If you want to learn more about how Workflows works, how you can use it to build applications, and how we built it, read on.

\n
\n

Workflows? Durable Execution?

\n \n \n \n
\n

Workflows—which we announced back during Developer Week earlier this year—is our take on the concept of “Durable Execution”: the ability to build and execute applications that are durable in the face of errors, network issues, upstream API outages, rate limits, and (most importantly) infrastructure failure.

As over 2.4 million developers continue to build applications on top of Cloudflare Workers, R2, and Workers AI, we’ve noticed more developers building multi-step applications and workflows that process user data, transform unstructured data into structured, export metrics, persist state as they progress, and automatically retry & restart. But writing any non-trivial application and making it durable in the face of failure is hard: this is where Workflows comes in. Workflows manages the retries, emitting the metrics, and durably storing the state (without you having to stand up your own database) as the Workflow progresses.

What makes Workflows different from other takes on “Durable Execution” is that we manage the underlying compute and storage infrastructure for you. You’re not left managing a compute cluster and hoping it scales both up (on a Monday morning) and down (during quieter periods) to manage costs, or ensuring that you have compute running in the right locations. Workflows is built on Cloudflare Workers — our job is to run your code and operate the infrastructure for you.

As an example of how Workflows can help you build durable applications, assume you want to post-process file uploads from your users that were uploaded to an R2 bucket directly via a pre-signed URL. That post-processing could involve multiple actions: text extraction via a Workers AI model, calls to a third-party API to validate data, updating or querying rows in a database once the file has been processed… the list goes on.

But what each of these actions has in common is that it could fail. Maybe that upstream API is unavailable, maybe you get rate-limited, maybe your database is down. Having to write extensive retry logic around each action, manage backoffs, and (importantly) ensure your application doesn’t have to start from scratch when a later step fails is more boilerplate to write and more code to test and debug.

What’s a step, you ask? The core building block of every Workflow is the step: an individually retriable component of your application that can optionally emit state. That state is then persisted, even if subsequent steps were to fail. This means that your application doesn’t have to restart, allowing it to not only recover more quickly from failure scenarios, but it can also avoid doing redundant work. You don’t want your application hammering an expensive third-party API (or getting you rate limited) because it’s naively retrying an API call that you don’t have to.

\n 
export class MyWorkflow extends WorkflowEntrypoint<Env, Params> {\n\tasync run(event: WorkflowEvent<Params>, step: WorkflowStep) {\n\t\tconst files = await step.do('my first step', async () => {\n\t\t\treturn {\n\t\t\t\tinputParams: event,\n\t\t\t\tfiles: [\n\t\t\t\t\t'doc_7392_rev3.pdf',\n\t\t\t\t\t'report_x29_final.pdf',\n\t\t\t\t\t'memo_2024_05_12.pdf',\n\t\t\t\t\t'file_089_update.pdf',\n\t\t\t\t\t'proj_alpha_v2.pdf',\n\t\t\t\t\t'data_analysis_q2.pdf',\n\t\t\t\t\t'notes_meeting_52.pdf',\n\t\t\t\t\t'summary_fy24_draft.pdf',\n\t\t\t\t],\n\t\t\t};\n\t\t});\n\n\t\t// Other steps...\n\t}\n}\n
\n

Notably, a Workflow can have hundreds of steps: one of the Rules of Workflows is to encapsulate every API call or stateful action within your application into its own step. Each step can also define its own retry strategy, automatically backing off, adding a delay and/or (eventually) giving up after a set number of attempts.

\n 
await step.do(\n\t'make a call to write that could maybe, just might, fail',\n\t// Define a retry strategy\n\t{\n\t\tretries: {\n\t\t\tlimit: 5,\n\t\t\tdelay: '5 seconds',\n\t\t\tbackoff: 'exponential',\n\t\t},\n\t\ttimeout: '15 minutes',\n\t},\n\tasync () => {\n\t\t// Do stuff here, with access to the state from our previous steps\n\t\tif (Math.random() > 0.5) {\n\t\t\tthrow new Error('API call to $STORAGE_SYSTEM failed');\n\t\t}\n\t},\n);\n
\n

To illustrate this further, imagine you have an application that reads text files from an R2 storage bucket, pre-processes the text into chunks, generates text embeddings using Workers AI, and then inserts those into a vector database (like Vectorize) for semantic search.

\n
\n \n

In the Workflows programming model, each of those is a discrete step, and each can emit state. For example, each of the four actions below can be a discrete step.do call in a Workflow:

  1. Reading the files from storage and emitting the list of filenames

  2. Chunking the text and emitting the results

  3. Generating text embeddings

  4. Upserting them into Vectorize and capturing the result of a test query

You can also start to imagine that some steps, such as chunking text or generating text embeddings, can be broken down into even more steps — a step per file that we chunk, or a step per API call to our text embedding model, so that our application is even more resilient to failure.

Steps can be created programmatically or conditionally based on input, allowing you to dynamically create steps based on the number of inputs your application needs to process. You do not need to define all steps ahead of time, and each instance of a Workflow may choose to conditionally create steps on the fly.

\n
\n

Building Cloudflare on Cloudflare

\n \n \n \n
\n

As the Cloudflare Developer platform continues to grow, almost all of our own products are built on top of it. Workflows is yet another example of how we built a new product from scratch using nothing but Workers and its vast catalog of features and APIs. This section of the blog has two goals: to explain how we built it, and to demonstrate that anyone can create a complex application or platform with demanding requirements and multiple architectural layers on our stack, too.

If you’re wondering how Workflows manages to make durable execution easy, how it persists state, and how it automatically scales: it’s because we built it on Cloudflare Workers, including the brand-new zero-latency SQLite storage we recently introduced to Durable Objects.\n

To understand how Workflows uses Workers & Durable Objects, here’s the high-level overview of our architecture:

\n
\n \n

There are three main blocks in this diagram:

The user-facing APIs are where the user interacts with the platform, creating and deploying new workflows or instances, controlling them, and accessing their state and activity logs. These operations can be executed through our public API gateway using REST calls, a Worker script using bindings, Wrangler (Cloudflare's developer platform command line tool), or via the Dashboard user interface.

The managed platform holds the internal configuration APIs running on a Worker implementing a catalog of REST endpoints, the binding shim, which is supported by another dedicated Worker, every account controller, and their correspondent workflow engines, all powered by SQLite-backed Durable Objects. This is where all the magic happens and what we are sharing more details about in this technical blog.

Finally, there are the workflow instances, essentially independent clones of the workflow application. Instances are user account-owned and have a one-to-one relationship with a managed engine that powers them. You can run as many instances and engines as you want concurrently.

Let's get into more detail…

\n
\n

Configuration API and Binding Shim

\n \n \n \n
\n \n
\n \n

The Configuration API and the Binding Shim are two stateless Workers; one receives REST API calls from clients calling our API Gateway directly, using Wrangler, or navigating the Dashboard UI, and the other is the endpoint for the Workflows binding, an efficient and authenticated interface to interact with the Cloudflare Developer Platform resources from a Workers script.

The configuration API worker uses HonoJS and Zod to implement the REST endpoints, which are declared in an OpenAPI schema and exported to our API Gateway, thus adding our methods to the Cloudflare API catalog.

\n 
import { swaggerUI } from '@hono/swagger-ui';\nimport { createRoute, OpenAPIHono, z } from '@hono/zod-openapi';\nimport { Hono } from 'hono';\n\n...\n\n​​api.openapi(\n  createRoute({\n    method: 'get',\n    path: '/',\n    request: {\n      query: PaginationParams,\n    },\n    responses: {\n      200: {\n        content: {\n          'application/json': {\n             schema: APISchemaSuccess(z.array(WorkflowWithInstancesCountSchema)),\n          },\n        },\n        description: 'List of all Workflows belonging to a account.',\n      },\n    },\n  }),\n  async (ctx) => {\n    ...\n  },\n);\n\n...\n\napi.route('/:workflow_name', routes.workflows);\napi.route('/:workflow_name/instances', routes.instances);\napi.route('/:workflow_name/versions', routes.versions);
\n

These Workers perform two different functions, but they share a large portion of their code and implement similar logic; once the request is authenticated and ready to travel to the next stage, they use the account ID to delegate the operation to a Durable Object called Account Controller.

\n 
// env.ACCOUNTS is the Account Controllers Durable Objects namespace\nconst accountStubId = c.env.ACCOUNTS.idFromName(accountId.toString());\nconst accountStub = c.env.ACCOUNTS.get(accountStubId);
\n

As you can see, every account has its own Account Controller Durable Object.

\n
\n

Account Controllers

\n \n \n \n
\n

The Account Controller is a dedicated persisted database that stores the list of all the account’s workflows, versions, and instances. We scale to millions of account controllers, one per every Cloudflare account using Workflows, by leveraging the power of Durable Objects with SQLite backend.

Durable Objects (DOs) are single-threaded singletons that run in our data centers and are bound to a stateful storage API, in this case, SQLite. They are also Workers, just a special kind, and have access to all of our other APIs. This makes it easy to build consistent, highly available distributed applications with them.

Here’s what we get for free by using one Durable Object per Workflows account:

Before SQLite, our only option was to use the Durable Object's key-value storage API, but having a relational database at our fingertips and being able to create tables and do complex queries is a significant enabler. For example, take a look at how we implement the internal method getWorkflow():

\n 
async function getWorkflow(accountId: number, workflowName: string) {\n  try {\n    const res = this.ctx.storage.transactionSync(() => {\n      const cursor = Array.from(\n        this.ctx.storage.sql.exec(\n          `\n                    SELECT *,\n                    (SELECT class_name\n                        FROM   versions\n                        WHERE  workflow_id = w.id\n                        ORDER  BY created_on DESC\n                        LIMIT  1) AS class_name\n                    FROM   workflows w\n                    WHERE  w.name = ? \n                    `,\n          workflowName\n        )\n      )[0] as Workflow;\n\n      return cursor;\n    });\n\n    this.sendAnalytics(accountId, begin, \"getWorkflow\");\n    return res as Workflow | undefined;\n  } catch (err) {\n    this.sendErrorAnalytics(accountId, begin, \"getWorkflow\");\n    throw err;\n  }\n}\n
\n

The other thing we take advantage of in Workflows is using the recently announced JavaScript-native RPC feature when communicating between components.

Before RPC, we had to fetch() between components, make HTTP requests, and serialize and deserialize the parameters and the payload. Now, we can async call the remote object's method as if it was local. Not only does this feel more natural and simplify our logic, but it's also more efficient, and we can take advantage of TypeScript type-checking when writing code.

This is how the Configuration API would call the Account Controller’s countWorkflows() method before:

\n 
const resp = await accountStub.fetch(\n      \"https://controller/count-workflows\",\n      {\n        method: \"POST\",\n        headers: {\n          \"Content-Type\": \"application/json; charset=utf-8\",\n        },\n        body: JSON.stringify({ accountId }),\n      },\n    );\n\nif (!resp.ok) {\n  return new Response(\"Internal Server Error\", { status: 500 });\n}\n\nconst result = await resp.json();\nconst total_count = result.total_count;
\n

This is how we do it using RPC:

\n 
const total_count = await accountStub.countWorkflows(accountId);
\n

The other powerful feature of our RPC system is that it supports passing not only Structured Cloneable objects back and forth but also entire classes. More on this later.

Let’s move on to Engine.

\n
\n

Engine and instance

\n \n \n \n
\n

Every instance of a workflow runs alongside an Engine instance. The Engine is responsible for starting up the user’s workflow entry point, executing the steps on behalf of the user, handling their results, and tracking the workflow state until completion.

\n
\n \n

When we started thinking about the Engine, we thought about modeling it after a state machine, and that was what our initial prototypes looked like. However, state machines require an ahead-of-time understanding of the userland code, which implies having a build step before running them. This is costly at scale and introduces additional complexity.

A few iterations later, we had another idea. What if we could model the engine as a game loop?

Unlike other computer programs, games operate regardless of a user's input. The game loop is essentially a sequence of tasks that implement the game's logic and update the display, typically one loop per video frame. Here’s an example of a game loop in pseudo-code:

\n 
while (game in running)\n    check for user input\n    move graphics\n    play sounds\nend while
\n

Well, an oversimplified version of our Workflow engine would look like this:

\n 
while (last step not completed)\n    iterate every step\n       use memoized cache as response if the step has run already\n       continue running step or timer if it hasn't finished yet\nend while
\n

A workflow is indeed a loop that keeps on going, performing the same sequence of logical tasks until the last step completes.

The Engine and the instance run hand-in-hand in a one-to-one relationship. The first is managed, and part of the platform. It uses SQLite and other platform APIs internally, and we can constantly add new features, fix bugs, and deploy new versions, while keeping everything transparent to the end user. The second is the actual account-owned Worker script that declares the Workflow steps.

For example, when someone passes a callback into step.do():

\n 
export class MyWorkflow extends WorkflowEntrypoint<Env, Params> {\n  async run(event: WorkflowEvent<Params>, step: WorkflowStep) {\n    step.do('step1', () => { ... });\n  }\n}
\n

We switch execution over to the Engine. Again, this is possible because of the power of JS RPC. Besides passing Structured Cloneable objects back and forth, JS RPC allows us to create and pass entire application-defined classes that extend the built-in RpcTarget. So this is what happens behind the scenes when your Instance calls step.do() (simplified):

\n 
export class Context extends RpcTarget {\n\n  async do<T>(name: string, callback: () => Promise<T>): Promise<T> {\n\n    // First we check we have a cache of this step.do() already\n    const maybeResult = await this.#state.storage.get(name);\n\n    // We return the cache if it exists\n    if (maybeValue) { return maybeValue; }\n\n    // Else we run the user callback\n    return doWrapper(callback);\n  }\n\n}\n
\n

Here’s a more complete diagram of the Engine’s step.do() lifecycle:

\n
\n \n

Again, this diagram only partially represents everything we do in the Engine; things like logging for observability or handling exceptions are missing, and we don't get into the details of how queuing is implemented. However, it gives you a good idea of how the Engine abstracts and handles all the complexities of completing a step under the hood, allowing us to expose a simple-to-use API to end users.

Also, it's worth reiterating that every workflow instance is an Engine behind the scenes, and every Engine is an SQLite-backed Durable Object. This ensures that every instance runtime and state are isolated and independent of each other and that we can effortlessly scale to run billions of workflow instances, a solved problem for Durable Objects.

\n
\n \n
\n
\n

Durability

\n \n \n \n
\n

Durable Execution is all the rage now when we talk about workflow engines, and ours is no exception. Workflows are typically long-lived processes that run multiple functions in sequence where anything can happen. Those functions can time out or fail because of a remote server error or a network issue and need to be retried. A workflow engine ensures that your application runs smoothly and completes regardless of the problems it encounters.

Durability means that if and when a workflow fails, the Engine can re-run it, resume from the last recorded step, and deterministically re-calculate the state from all the successful steps' cached responses. This is possible because steps are stateful and idempotent; they produce the same result no matter how many times we run them, thus not causing unintended duplicate effects like sending the same invoice to a customer multiple times.

\n
\n \n

We ensure durability and handle failures and retries by sharing the same technique we use for a step.sleep() that requires sleeping for days or months: a combination of using scheduler.wait(), a method of the upcoming WICG Scheduling API that we already support, and Durable Objects alarms, which allow you to schedule the Durable Object to be woken up at a time in the future.

These two APIs allow us to overcome the lack of guarantees that a Durable Object runs forever, giving us complete control of its lifecycle. Since every state transition through userland code persists in the Engine’s strongly consistent SQLite, we track timestamps when a step begins execution, its attempts (if it needs retries), and its completion.

\n
\n \n

This means that steps pending if a Durable Object is evicted — perhaps due to a two-month-long timer — get rerun on the next lifetime of the Engine (with its cache from the previous lifetime hydrated) that is triggered by an alarm set with the timestamp of the next expected state transition. 

\n
\n

Real-life workflow, step by step

\n \n \n \n
\n

Let's walk through an example of a real-life application. You run an e-commerce website and would like to send email reminders to your customers for forgotten carts that haven't been checked out in a few days.

What would typically have to be a combination of a queue, a cron job, and querying a database table periodically can now simply be a Workflow that we start on every new cart:

\n 
import {\n  WorkflowEntrypoint,\n  WorkflowEvent,\n  WorkflowStep,\n} from \"cloudflare:workers\";\nimport { sendEmail } from \"./legacy-email-provider\";\n\ntype Params = {\n  cartId: string;\n};\n\ntype Env = {\n  DB: D1Database;\n};\n\nexport class Purchase extends WorkflowEntrypoint<Env, Params> {\n  async run(\n    event: WorkflowEvent<Params>,\n    step: WorkflowStep\n  ): Promise<unknown> {\n    await step.sleep(\"wait for three days\", \"3 days\");\n\n    // Retrieve cart from D1\n    const cart = await step.do(\"retrieve cart from database\", async () => {\n      const { results } = await this.env.DB.prepare(`SELECT * FROM cart WHERE id = ?`)\n        .bind(event.payload.cartId)\n        .all();\n      return results[0];\n    });\n\n    if (!cart.checkedOut) {\n      await step.do(\"send an email\", async () => {\n        await sendEmail(\"reminder\", cart);\n      });\n    }\n  }\n}\n
\n

This works great. However, sometimes the sendEmail function fails due to an upstream provider erroring out. While step.do automatically retries with a reasonable default configuration, we can define our settings:

\n 
if (cart.isComplete) {\n  await step.do(\n    \"send an email\",\n    {\n      retries: {\n        limit: 5,\n        delay: \"1 min\",\n        backoff: \"exponential\",\n      },\n    },\n    async () => {\n      await sendEmail(\"reminder\", cart);\n    }\n  );\n}\n
\n \n
\n

Managing Workflows

\n \n \n \n
\n

Workflows allows us to create and manage workflows using four different interfaces:

The HTTP API makes it easy to trigger new instances of workflows from any system, even if it isn’t on Cloudflare, or from the command line. For example:

\n 
curl --request POST \\\n  --url https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/workflows/purchase-workflow/instances/$CART_INSTANCE_ID \\\n  --header 'Authorization: Bearer $ACCOUNT_TOKEN \\\n  --header 'Content-Type: application/json' \\\n  --data '{\n\t\"id\": \"$CART_INSTANCE_ID\",\n\t\"params\": {\n\t\t\"cartId\": \"f3bcc11b-2833-41fb-847f-1b19469139d1\"\n\t}\n  }'
\n

Wrangler goes one step further and gives us a friendlier set of commands to interact with workflows with fancy formatted outputs without needing to authenticate with tokens. Type npx wrangler workflows for help, or:

\n 
npx wrangler workflows trigger purchase-workflow '{ \"cartId\": \"f3bcc11b-2833-41fb-847f-1b19469139d1\" }'
\n

Furthermore, Workflows has first-party support in wrangler, and you can test your instances locally. A Workflow is similar to a regular WorkerEntrypoint in your Worker, which means that wrangler dev just naturally works.

\n 
❯ npx wrangler dev\n\n ⛅️ wrangler 3.82.0\n----------------------------\n\nYour worker has access to the following bindings:\n- Workflows:\n  - CART_WORKFLOW: EcommerceCartWorkflow\n⎔ Starting local server...\n[wrangler:inf] Ready on http://localhost:8787\n╭───────────────────────────────────────────────╮\n│  [b] open a browser, [d] open devtools        │\n╰───────────────────────────────────────────────╯\n
\n

Workflow APIs are also available as a Worker binding. You can interact with the platform programmatically from another Worker script in the same account without worrying about permissions or authentication. You can even have workflows that call and interact with other workflows.

\n 
import { WorkerEntrypoint } from \"cloudflare:workers\";\n\ntype Env = { DEMO_WORKFLOW: Workflow };\nexport default class extends WorkerEntrypoint<Env> {\n  async fetch() {\n    // Pass in a user defined name for this instance\n    // In this case, we use the same as the cartId\n    const instance = await this.env.DEMO_WORKFLOW.create({\n      id: \"f3bcc11b-2833-41fb-847f-1b19469139d1\",\n      params: {\n          cartId: \"f3bcc11b-2833-41fb-847f-1b19469139d1\",\n      }\n    });\n  }\n  async scheduled() {\n    // Restart errored out instances in a cron\n    const instance = await this.env.DEMO_WORKFLOW.get(\n      \"f3bcc11b-2833-41fb-847f-1b19469139d1\"\n    );\n    const status = await instance.status();\n    if (status.error) {\n      await instance.restart();\n    }\n  }\n}
\n \n
\n

Observability 

\n \n \n \n
\n

Having good observability and data on often long-lived asynchronous tasks is crucial to understanding how we're doing under normal operation and, more importantly, when things go south, and we need to troubleshoot problems or when we are iterating on code changes.

We designed Workflows around the philosophy that there is no such thing as too much logging. You can get all the SQLite data for your workflow and its instances by calling the REST APIs. Here is the output of an instance:

\n 
{\n  \"success\": true,\n  \"errors\": [],\n  \"messages\": [],\n  \"result\": {\n    \"status\": \"running\",\n    \"params\": {},\n    \"trigger\": { \"source\": \"api\" },\n    \"versionId\": \"ae042999-39ff-4d27-bbcd-22e03c7c4d02\",\n    \"queued\": \"2024-10-21 17:15:09.350\",\n    \"start\": \"2024-10-21 17:15:09.350\",\n    \"end\": null,\n    \"success\": null,\n    \"steps\": [\n      {\n        \"name\": \"send email\",\n        \"start\": \"2024-10-21 17:15:09.411\",\n        \"end\": \"2024-10-21 17:15:09.678\",\n        \"attempts\": [\n          {\n            \"start\": \"2024-10-21 17:15:09.411\",\n            \"end\": \"2024-10-21 17:15:09.678\",\n            \"success\": true,\n            \"error\": null\n          }\n        ],\n        \"config\": {\n          \"retries\": { \"limit\": 5, \"delay\": 1000, \"backoff\": \"constant\" },\n          \"timeout\": \"15 minutes\"\n        },\n        \"output\": \"celso@example.com\",\n        \"success\": true,\n        \"type\": \"step\"\n      },\n      {\n        \"name\": \"sleep-1\",\n        \"start\": \"2024-10-21 17:15:09.763\",\n        \"end\": \"2024-10-21 17:17:09.763\",\n        \"finished\": false,\n        \"type\": \"sleep\",\n        \"error\": null\n      }\n    ],\n    \"error\": null,\n    \"output\": null\n  }\n}
\n

As you can see, this is essentially a dump of the instance engine SQLite in JSON. You have the errors, messages, current status, and what happened with every step, all time stamped to the millisecond.

It's one thing to get data about a specific workflow instance, but it's another to zoom out and look at aggregated statistics of all your workflows and instances over time. Workflows data is available through our GraphQL Analytics API, so you can query it in aggregate and generate valuable insights and reports. In this example we ask for aggregated analytics about the wall time of all the instances of the “e-commerce-carts” workflow:

\n 
{\n  viewer {\n    accounts(filter: { accountTag: \"febf0b1a15b0ec222a614a1f9ac0f0123\" }) {\n      wallTime: workflowsAdaptiveGroups(\n        limit: 10000\n        filter: {\n          datetimeHour_geq: \"2024-10-20T12:00:00.000Z\"\n          datetimeHour_leq: \"2024-10-21T12:00:00.000Z\"\n          workflowName: \"e-commerce-carts\"\n        }\n        orderBy: [count_DESC]\n      ) {\n        count\n        sum {\n          wallTime\n        }\n        dimensions {\n          date: datetimeHour\n        }\n      }\n    }\n  }\n}\n
\n

For convenience, you can evidently also use Wrangler to describe a workflow or an instance and get an instant and beautifully formatted response:

\n 
sid ~ npx wrangler workflows instances describe purchase-workflow latest\n\n ⛅️ wrangler 3.80.4\n\nWorkflow Name:         purchase-workflow\nInstance Id:           d4280218-7756-41d2-bccd-8d647b82d7ce\nVersion Id:            0c07dbc4-aaf3-44a9-9fd0-29437ed11ff6\nStatus:                ✅ Completed\nTrigger:               🌎 API\nQueued:                14/10/2024, 16:25:17\nSuccess:               ✅ Yes\nStart:                 14/10/2024, 16:25:17\nEnd:                   14/10/2024, 16:26:17\nDuration:              1 minute\nLast Successful Step:  wait for three days\nOutput:                false\nSteps:\n\n  Name:      wait for three days\n  Type:      💤 Sleeping\n  Start:     14/10/2024, 16:25:17\n  End:       17/10/2024, 16:25:17\n  Duration:  3 day
\n

And finally, we worked really hard to get you the best dashboard UI experience when navigating Workflows data.

\n
\n \n
\n
\n

So, how much does it cost?

\n \n \n \n
\n

It’d be painful if we introduced a powerful new way to build Workers applications but made it cost prohibitive.

Workflows is priced just like Cloudflare Workers, where we introduced CPU-based pricing: only on active CPU time and requests, not duration (aka: wall time).

\n
\n \n

Workers Standard pricing model

This is especially advantageous when building the long-running, multi-step applications that Workflows enables: if you had to pay while your Workflow was sleeping, waiting on an event, or making a network call to an API, writing the “right” code would be at odds with writing affordable code.

There’s also no need to keep a Kubernetes cluster or a group of virtual machines running (and burning a hole in your wallet): we manage the infrastructure, and you only pay for the compute your Workflows consume.   

\n
\n

What’s next?

\n \n \n \n
\n

Today, after months of developing the platform, we are announcing the open beta program, and we couldn't be more excited to see how you will be using Workflows. Looking forward, we want to do things like triggering instances from queue messages and have other ideas, but at the same time, we are certain that your feedback will help us shape the roadmap ahead.

We hope that this blog post gets you thinking about how to use Workflows for your next application, but also that it inspires you on what you can build on top of Workers. Workflows as a platform is entirely built on top of Workers, its resources, and APIs. Anyone can do it, too.

To chat with the team and other developers building on Workflows, join the #workflows-beta channel on the Cloudflare Developer Discord, and keep an eye on the Workflows changelog during the beta. Otherwise, visit the Workflows tutorial to get started.

If you're an engineer, look for opportunities to work with us and help us improve Workflows or build other products.

", "url": "https://blog.cloudflare.com/building-workflows-durable-execution-on-workers/", "title": "Build durable applications on Cloudflare Workers: you write the Workflows, we take care of the rest", "date_modified": "2024-10-24T13:00:00.000Z" }, { "content_html": "

With the rapid advancements occurring in the AI space, developers face significant challenges in keeping up with the ever-changing landscape. New models and providers are continuously emerging, and understandably, developers want to experiment and test these options to find the best fit for their use cases. This creates the need for a streamlined approach to managing multiple models and providers, as well as a centralized platform to efficiently monitor usage, implement controls, and gather data for optimization.

AI Gateway is specifically designed to address these pain points. Since its launch in September 2023, AI Gateway has empowered developers and organizations by successfully proxying over 2 billion requests in just one year, as we highlighted during September’s Birthday Week. With AI Gateway, developers can easily store, analyze, and optimize their AI inference requests and responses in real time.

With our initial architecture, AI Gateway faced a significant challenge: the logs, those critical trails of data interactions between applications and AI models, could only be retained for 30 minutes. This limitation was not just a minor inconvenience; it posed a substantial barrier for developers and businesses needing to analyze long-term patterns, ensure compliance, or simply debug over more extended periods.

In this post, we'll explore the technical challenges and strategic decisions behind extending our log storage capabilities from 30 minutes to being able to store billions of logs indefinitely. We'll discuss the challenges of scale, the intricacies of data management, and how we've engineered a system that not only meets the demands of today, but is also scalable for the future of AI development.

\n
\n

Background

\n \n \n \n
\n

AI Gateway is built on Cloudflare Workers, a serverless platform that runs on the Cloudflare network, allowing developers to write small JavaScript functions that can execute at the point of need, near the user, on Cloudflare's vast network of data centers, without worrying about platform scalability.

\n
\n \n

Our customers use multiple providers and models and are always looking to optimize the way they do inference. And, of course, in order to evaluate their prompts, performance, cost, and to troubleshoot what’s going on, AI Gateway’s customers need to store requests and responses. New requests show up within 15 seconds and customers can check a request’s cost, duration, number of tokens, and provide their feedback (thumbs up or down).

\n
\n \n

This scales in a way where an account can have multiple gateways and each gateway has its own settings. In our first implementation, a backend worker was responsible for storing Real Time Logs and other background tasks. However, in the rapidly evolving domain of artificial intelligence, where real-time data is as precious as the insights it provides, managing log data efficiently becomes paramount. We recognized that to truly empower our users, we needed to offer a solution where logs weren't just transient records but could be stored permanently. Permanent log storage means developers can now track the performance, security, and operational insights of their AI applications over time, enabling not only immediate troubleshooting but also longitudinal studies of AI behavior, usage trends, and system health.

\n
\n \n

The diagram above describes our old architecture, which could only store 30 minutes of data.

Tracing the path of a request through the AI Gateway, as depicted in the sequence above:

  1. A developer sends a new inference request, which is first received by our Gateway Worker.

  2. The Gateway Worker then performs several checks: it looks for cached results, enforces rate limits, and verifies any other configurations set by the user for their gateway. Provided all conditions are met, it forwards the request to the selected inference provider (in this diagram, OpenAI).

  3. The inference provider processes the request and sends back the response.

  4. Simultaneously, as the response is relayed back to the developer, the request and response details are also dispatched to our Backend Worker. This worker's role is to manage and store the log of this transaction.

\n
\n

The challenge: Store two billion logs

\n \n \n \n
\n \n
\n

First step: real-time logs

\n \n \n \n
\n

Initially, the AI Gateway project stored both request metadata and the actual request bodies in a D1 database. This approach facilitated rapid development in the project's infancy. However, as customer engagement grew, the D1 database began to fill at an accelerating rate, eventually retaining logs for only 30 minutes at a time.

To mitigate this, we first optimized the database schema, which extended the log retention to one hour. However, we soon encountered diminishing returns due to the sheer volume of byte data from the request bodies. Post-launch, it became clear that a more scalable solution was necessary. We decided to migrate the request bodies to R2 storage, significantly alleviating the data load on D1. This adjustment allowed us to incrementally extend log retention to 24 hours.

Consequently, D1 functioned primarily as a log index, enabling users to search and filter logs efficiently. When users needed to view details or download a log, these actions were seamlessly proxied through to R2.

This dual-system approach provided us with the breathing room to contemplate and develop more sophisticated storage solutions for the future.

\n
\n

Second step: persistent logs and Durable Object transactional storage

\n \n \n \n
\n

As our traffic surged, we encountered a growing number of requests from customers wanting to access and compare older logs.

Upon learning that the Durable Objects team was seeking beta testers for their new Durable Objects with SQLite, we eagerly signed up.

Originally, we considered Durable Objects as the ideal solution for expanding our log storage capacity, which required us to shard the logs by a unique string. Initially, this string was the account ID, but during a mid-development load test, we hit a cap at 10 million logs per Durable Object. This limitation meant that each account could only support up to this number of logs.

Given our commitment to the DO migration, we saw an opportunity rather than a constraint. To overcome the 10 million log limit per account, we refined our approach to shard by both account ID and gateway name. This adjustment effectively raised the storage ceiling from 10 million logs per account to 10 million per gateway. With the default setting allowing each account up to 10 gateways, the potential storage for each account skyrocketed to 100 million logs.

This strategic pivot not only enabled us to store a significantly larger number of logs. But also enhanced our flexibility in gateway management. Now, when a gateway is deleted, we can simply remove the corresponding Durable Object.

Additionally, this sharding method isolates high-volume request scenarios. If one customer's heavy usage slows down log insertion, it only impacts their specific Durable Object, thereby preserving performance for other customers.

\n
\n \n

Taking a glance at the revised architecture diagram, we replaced the Backend Worker with our newly integrated Durable Object. The rest of the request flow remains unchanged, including the concurrent response to the user and the interaction with the Durable Object, which occurs in the fourth step.

Leveraging Cloudflare’s network, our Gateway Worker operates near the user's location, which in turn positions the user's Durable Object close by. This proximity significantly enhances the speed of log insertion and query operations.

\n
\n

Third step: managing thousands of Durable Objects

\n \n \n \n
\n

As the number of users and requests on AI Gateway grows, managing each unique Durable Object (DO) becomes increasingly complex. New customers join continuously, and we needed an efficient method to track each DO, ensure users stay within their 10 gateway limit, and manage the storage capacity for free users.

To address these challenges, we introduced another layer of control with a new Durable Object we've named the Account Manager. The primary function of the Account Manager is straightforward yet crucial: it keeps user activities in check.

Here's how it works: before any Gateway commits a new log to permanent storage, it consults the Account Manager. This check determines whether the gateway is allowed to insert the log based on the user's current usage and entitlements. The Account Manager uses its own SQLite database to verify the total number of rows a user has and their service level. If all checks pass, it signals the Gateway that the log can be inserted. It was paramount to guarantee that this entire validation process occurred in the background, ensuring that the user experience remains seamless and uninterrupted.

The Account Manager stays updated by periodically receiving data from each Gateway’s Durable Object. Specifically, after every 1000 inference requests, the Gateway sends an update on its total rows to the Account Manager, which then updates its local records. This system ensures that the Account Manager has the most current data when making its decisions.

Additionally, the Account Manager is responsible for monitoring customer entitlements. It tracks whether an account is on a free or paid plan, how many gateways a user is permitted to create, and the log storage capacity allocated to each gateway. 

Through these mechanisms, the Account Manager not only helps in maintaining system integrity but also ensures fair usage across all users of AI Gateway.

\n
\n

AI evaluations and Durable Objects sharding

\n \n \n \n
\n

As we continue to develop evaluations to fully automatic and, in the future, use Large Language Models (LLMs),  we are now taking the first step towards this goal and launching the open beta phase of comprehensive AI evaluations, centered on Human-in-the-Loop feedback.

This feature empowers users to create bespoke datasets from their application logs, thereby enabling them to score and evaluate the performance, speed, and cost-effectiveness of their models, with a primary focus on LLMs and automated scoring, analyzing the performance of LLMs, providing developers with objective, data-driven insights to refine their models.

To do this, developers require a reliable logging mechanism that persists logs from multiple gateways, storing up to 100 million logs in total (10 million logs per gateway, across 10 gateways). This represents a significant volume of data, as each request made through the AI Gateway generates a log entry, with some log entries potentially exceeding 50 MB in size.

This necessity leads us to work on the expansion of log storage capabilities. Since log storage is limited to 10 million logs per gateway, in future iterations, we aim to scale this capacity by implementing sharded Durable Objects (DO), allowing multiple Durable Objects per gateway to handle and store logs. This scaling strategy will enable us to store significantly larger volumes of logs, providing richer data for evaluations (using LLMs as a judge or from user input), all through AI Gateway.

\n
\n \n
\n
\n

Coming Soon

\n \n \n \n
\n

We are working on improving our existing Universal Endpoint, the next step on an enhanced solution that builds on existing fallback mechanisms to offer greater resilience, flexibility, and intelligence in request management.

Currently, when a provider encounters an error or is unavailable, our system falls back to an alternative provider to ensure continuity. The improved Universal Endpoint takes this a step further by introducing automatic retry capabilities, allowing failed requests to be reattempted before fallback is triggered. This significantly improves reliability by handling transient errors and increasing the likelihood of successful request fulfillment. It will look something like this:

\n 
curl --location 'https://aig.example.com/' \\\n--header 'CF-AIG-TOKEN: Bearer XXXX' \\\n--header 'Content-Type: application/json' \\\n--data-raw '[\n    {\n        \"id\": \"0001\",\n        \"provider\": \"openai\",\n        \"endpoint\": \"chat/completions\",\n        \"headers\": {\n            \"Authorization\": \"Bearer XXXX\",\n            \"Content-Type\": \"application/json\"\n        },\n        \"query\": {\n            \"model\": \"gpt-3.5-turbo\",\n            \"messages\": [\n                {\n                    \"role\": \"user\",\n                    \"content\": \"generate a prompt to create cloudflare random images\"\n                }\n            ]\n        },\n        \"option\": {\n            \"retry\": 2,\n            \"delay\": 200,\n            \"onComplete\": {\n                \"provider\": \"workers-ai\",\n                \"endpoint\": \"@cf/stabilityai/stable-diffusion-xl-base-1.0\",\n                \"headers\": {\n                    \"Authorization\": \"Bearer A5UFQkHewHF1-sA3hTVQFaPxRuu5wmS0eJcCS_MC\",\n                    \"Content-Type\": \"application/json\"\n                },\n                \"query\": {\n                    \"messages\": [\n                        {\n                            \"role\": \"user\",\n                            \"content\": \"<prompt-response id='\\''0001'\\'' />\"\n                        }\n                    ]\n                }\n            }\n        }\n    },\n    {\n        \"provider\": \"workers-ai\",\n        \"endpoint\": \"@cf/stabilityai/stable-diffusion-xl-base-1.0\",\n        \"headers\": {\n            \"Authorization\": \"Bearer XXXXXX\",\n            \"Content-Type\": \"application/json\"\n        },\n        \"query\": {\n            \"messages\": [\n                {\n                    \"role\": \"user\",\n                    \"content\": \"create a image of a missing cat\"\n                }\n            ]\n        }\n    }\n]'\n
\n

The request to the improved Universal Endpoint system demonstrates how it handles multiple providers with integrated retry mechanisms and fallback logic. In this example, the first request is sent to a provider like OpenAI, asking it to generate a text-to-image prompt. The “retry” option ensures that transient issues don’t result in immediate failure.

The system’s ability to seamlessly switch between providers while applying retry strategies ensures higher reliability and robustness in managing requests. By leveraging fallback logic, the Improved Universal Endpoint can dynamically adapt to provider failures, ensuring that tasks are completed successfully even in complex, multi-step workflows.

In addition to retry logic, we will have the ability to inspect requests and responses and make dynamic decisions based on the content of the result. This enables developers to create conditional workflows where the system can adapt its behavior depending on the nature of the response, creating a highly flexible and intelligent decision-making process.

If you haven’t yet used AI Gateway, check out our developer documentation on how to get started. If you have any questions, reach out on our Discord channel.

", "url": "https://blog.cloudflare.com/billions-and-billions-of-logs-scaling-ai-gateway-with-the-cloudflare/", "title": "Billions and billions (of logs): scaling AI Gateway with the Cloudflare Developer Platform", "date_modified": "2024-10-24T13:00:00.000Z" }, { "content_html": "

Cloudflare Queues let a developer decouple their Workers into event-driven services. Producer Workers write events to a Queue, and consumer Workers are invoked to take actions on the events. For example, you can use a Queue to decouple an e-commerce website from a service which sends purchase confirmation emails to users. During 2024’s Birthday Week, we announced that Cloudflare Queues is now Generally Available, with significant performance improvements that enable larger workloads. To accomplish this, we switched to a new architecture for Queues that enabled the following improvements:

\n
\n \n

Median latency drops from ~200ms to ~60ms as Queues are migrated to the new architecture

In this blog post, we'll share details about how we built Queues using Durable Objects and the Cloudflare Developer Platform, and how we migrated from an initial Beta architecture to a geographically-distributed, horizontally-scalable architecture for General Availability.

\n
\n

v1 Beta architecture

\n \n \n \n
\n

When initially designing Cloudflare Queues, we decided to build something simple that we could get into users' hands quickly. First, we considered leveraging an off-the-shelf messaging system such as Kafka or Pulsar. However, we decided that it would be too challenging to operate these systems at scale with the large number of isolated tenants that we wanted to support.

Instead of investing in new infrastructure, we decided to build on top of one of Cloudflare's existing developer platform building blocks: Durable Objects. Durable Objects are a simple, yet powerful building block for coordination and storage in a distributed system. In our initial v1 architecture, each Queue was implemented using a single Durable Object. As shown below, clients would send messages to a Worker running in their region, which would be forwarded to the single Durable Object hosted in the WNAM (Western North America) region. We used a single Durable Object for simplicity, and hosted it in WNAM for proximity to our centralized configuration API service.

\n
\n \n

One of a Queue's main responsibilities is to accept and store incoming messages. Sending a message to a v1 Queue used the following flow:

Durable Objects handled most of the heavy-lifting here: we did not need to set up any new servers, storage, or service discovery infrastructure. To route requests, we simply provided a queueID and the platform handled the rest. To store messages, we used the Durable Object storage API to put each message, and the platform handled reliably storing the data redundantly.

\n
\n

Consuming messages

\n \n \n \n
\n

The other main responsibility of a Queue is to deliver messages to a Consumer. Delivering messages in a v1 Queue used the following process:

\n
\n \n

This v1 architecture let us flesh out the initial version of the Queues Beta product and onboard users quickly. Using Durable Objects allowed us to focus on building application logic, instead of complex low-level systems challenges such as global routing and guaranteed durability for storage. Using a separate Durable Object for each Queue allowed us to host an essentially unlimited number of Queues, and provided isolation between them.

However, using only one Durable Object per queue had some significant limitations:

To solve these issues, we created a new v2 architecture that horizontally scales across multiple Durable Objects to implement each single high-performance Queue.

\n
\n

v2 Architecture

\n \n \n \n
\n

In the new v2 architecture for Queues, each Queue is implemented using multiple Durable Objects, instead of just one. Instead of a single region, we place Storage Shard Durable Objects in all available regions to enable lower latency. Within each region, we create multiple Storage Shards and load balance incoming requests amongst them. Just like that, we’ve multiplied message throughput.

\n
\n \n

Sending a message to a v2 Queue uses the following flow:

In this v2 architecture, messages are stored in the closest available Durable Object storage cluster near the user, greatly reducing latency since messages don't need to be shipped all the way to WNAM. Using multiple shards within each region removes the bottleneck of a single Durable Object, and allows us to scale each Queue horizontally to accept even more messages per second. Workers KV acts as a fast metadata store: our Worker can quickly look up the shard map to perform load balancing across shards.

To improve the Consumer side of v2 Queues, we used a similar \"scale out\" approach. A single Durable Object can only perform a limited number of concurrent subrequests. In v1 Queues, this limited the number of concurrent subrequests we could make to our Dispatcher Worker. To work around this, we created a new Consumer Shard Durable Object class that we can scale horizontally, enabling us to execute many more concurrent instances of our users' queue() handlers.

\n
\n \n

Consumer Durable Objects in v2 Queues use the following approach:

This scale-out approach enabled us to work around the subrequest limits of a single Durable Object, and increase the maximum supported concurrency level of a Queue from 20 to 250. 

\n
\n

The Coordinator and “Control Plane”

\n \n \n \n
\n

So far, we have primarily discussed the \"Data Plane\" of a v2 Queue: how messages are load balanced amongst Storage Shards, and how Consumer Shards read and deliver messages. The other main piece of a v2 Queue is the \"Control Plane\", which handles creating and managing all the individual Durable Objects in the system. In our v2 architecture, each Queue has a single Coordinator Durable Object that acts as the brain of the Queue. Requests to create a Queue, or change its settings, are sent to the Queue's Coordinator.

\n
\n \n

The Coordinator maintains a Shard Map for the Queue, which includes metadata about all the Durable Objects in the Queue (including their region, number of available messages, current estimated load, etc.). The Coordinator periodically writes a fresh copy of the Shard Map into Workers KV, as pictured in step 1 of the diagram. Placing the shard map into Workers KV ensures that it is globally cached and available for our Worker to read quickly, so that it can pick a shard to accept the message.

Every shard in the system periodically sends a heartbeat to the Coordinator as shown in steps 2 and 3 of the diagram. Both Storage Shards and Consumer Shards send heartbeats, including information like the number of messages stored locally, and the current load (requests per second) that the shard is handling. The Coordinator uses this information to perform autoscaling. When it detects that the shards in a particular region are overloaded, it creates additional shards in the region, and adds them to the shard map in Workers KV. Our Worker sees the updated shard map and naturally load balances messages across the freshly added shards. Similarly, the Coordinator looks at the backlog of available messages in the Queue, and decides to add more Consumer shards to increase Consumer throughput when the backlog is growing. Consumer Shards pull messages from Storage Shards for processing as shown in step 4 of the diagram.

Switching to a new scalable architecture allowed us to meet our performance goals and take Queues to GA. As a recap, this new architecture delivered these significant improvements:

\n
\n

What's next for Queues

\n \n \n \n
\n

To start using Queues, head over to our Getting Started guide. 

Do distributed systems like Cloudflare Queues and Durable Objects interest you? Would you like to help build them at Cloudflare? We're Hiring!

", "url": "https://blog.cloudflare.com/how-we-built-cloudflare-queues", "title": "Durable Objects aren't just durable, they're fast: a 10x speedup for Cloudflare Queues", "date_modified": "2024-10-24T13:00:00.000Z" }, { "content_html": "Comments", "url": "https://benjdd.com/aws/", "title": "AWS data center latencies, visualized", "date_modified": "2024-10-24T03:18:23.000Z" }, { "content_html": "

I’m thrilled to announce macOS support in EC2 Image Builder. This new capability allows you to create and manage machine images for your macOS workloads in addition to the existing support for Windows and Linux.

\n

A golden image is a bootable disk image, also called an Amazon Machine Image (AMI), pre-installed with the operating system and all the tools required for your workloads. In the context of a continuous integration and continuous deployment (CI/CD) pipeline, your golden image most probably contains the specific version of your operating system (macOS) and all required development tools and libraries to build and test your applications (Xcode, Fastlane, and so on.)

\n

Developing and manually managing pipelines to build macOS golden images is time-consuming and diverts talented resources from other tasks. And when you have existing pipelines to build Linux or Windows images, you need to use different tools for creating macOS images, leading to a disjointed workflow.

\n

For these reasons, many of you have been asking for the ability to manage your macOS images using EC2 Image Builder. You want to consolidate your image pipelines across operating systems and take advantage of the automation and cloud-centered integrations that EC2 Image Builder provides.

\n

By adding macOS support to EC2 Image Builder, you can now streamline your image management processes and reduce the operational overhead of maintaining macOS images. EC2 Image Builder takes care of testing, versioning, and validating the base images at scale, saving you the costs associated with maintaining your preferred macOS versions.

\n

Let’s see it in action
Let’s create a pipeline to create a macOS AMI with Xcode 16. You can follow a similar process to install Fastlane on your AMIs.

\n

At a high level, there are four main steps.

\n
    \n
  1. I define a component for each tool I want to install. A component is a YAML document that tells EC2 Image Builder what application to install and how. In this example, I create a custom component to install Xcode. If you want to install Fastlane, you create a second component. I use the ExecuteBash action to enter the shell commands required to install Xcode.
    2. \n
  2. I define a recipe. A recipe starts from a base image and lists the components I want to install on it.
    3. \n
  3. I define the infrastructure configuration I want to use to build my image. This defines the pool of Amazon Elastic Compute Cloud (Amazon EC2) instances to build the image. In my case, I allocate an EC2 Mac Dedicated Host in my account and reference it in the infrastructure configuration.
    4. \n
  4. I create a pipeline and a schedule to run on the infrastructure with the given recipes and an image workflow. I test the output AMI and deliver it at the chosen destination (my account or another account)
    5. \n
\n

It’s much easier than it sounds. I’ll show you the steps in the AWS Management Console. I can also configure EC2 Image Builder with the AWS Command Line Interface (AWS CLI) or write code using one of our AWS SDKs.

\n

Step 1: Create a component
I open the console and select EC2 Image Builder, then Components, and finally Create component.

\n

\"Image

\n

I select a base Image operating system and the Compatible OS Versions. Then, I enter a Component name and Component version. I select Define document content and enter this YAML as Content.

\n
name: InstallXCodeDocument\ndescription: This downloads and installs Xcode. Be sure to run `xcodeinstall authenticate -s us-east-1` from your laptop first.\nschemaVersion: 1.0\n\nphases:\n  - name: build\n    steps:\n      - name: InstallXcode\n        action: ExecuteBash\n        inputs:\n          commands:\n             - sudo -u ec2-user /opt/homebrew/bin/brew tap sebsto/macos\n             - sudo -u ec2-user /opt/homebrew/bin/brew install xcodeinstall\n             - sudo -u ec2-user /opt/homebrew/bin/xcodeinstall download -s us-east-1 --name \"Xcode 16.xip\"\n             - sudo -u ec2-user /opt/homebrew/bin/xcodeinstall install --name \"Xcode 16.xip\"\n  \n  - name: validate\n    steps:\n      - name: TestXcode\n        action: ExecuteBash\n        inputs:\n          commands:\n            -  xcodebuild -version && xcode-select -p
\n

I use a tool I wrote to download and install Xcode from the command line. xcodeinstall integrates with AWS Secrets Manager to securely store authentication web tokens. Before running the pipeline, I authenticate from my laptop with the command xcodeinstall authenticate -s us-east-1. This command starts a session with Apple server’s and stores the session token in Secrets Manager. xcodeinstall uses this token during the image creation pipeline to download Xcode.

\n

When you use xcodeinstall with Secrets Manager, you must give permission to your pipeline to access the secrets. Here is the policy document I added to the role attached to the EC2 instance used by EC2 Image Builder (in the following infrastructure configuration).

\n
{\n\t\"Sid\": \"xcodeinstall\",\n\t\"Effect\": \"Allow\",\n\t\"Action\": [\n            \"secretsmanager:GetSecretValue\"\n            \"secretsmanager:PutSecretValue\"\n        ],\n\t\"Resource\": \"arn:aws:secretsmanager:us-east-1:<YOUR ACCOUNT ID>:secret:xcodeinstall*\"\n}
\n

To test and debug these components locally, without having to wait for long cycle to start and recycle the EC2 Mac instance, you can use the AWS Task Orchestrator and Executor (AWSTOE) command.

\n

Step 2: Create a recipe
The next step is to create a recipe. On the console, I select Image recipes and Create image recipe.

\n

I select macOS as the base Image Operating System. I choose macOS Sonoma ARM64 as Image name.

\n

In the Build components section, I select the Xcode 16 component I just created during step 1.

\n

Finally, I make sure the volume is large enough to store the operating system, Xcode, and my builds. I usually select a 500 Gb gp3 volume.

\n

\"Image

\n

Steps 3 and 4: Create the pipeline (and the infrastructure configuration)
On the EC2 Image Builder page, I select Image pipelines and Create image pipeline. I give my pipeline a name and select a Build schedule. For this demo, I select a manual trigger.\"Image

\n

Then, I select the recipe I just created (Sonoma-Xcode).

\n

\"Image

\n

I chose Default workflows for Define image creation process (not shown for brevity).

\n

I create or select an existing infrastructure configuration. In the context of building macOS images, you have to allocate Amazon EC2 Dedicated Hosts first. This is where I choose the instance type that EC2 Image Builder will use to create the AMI. I may also optionally select my virtual private cloud (VPC), security group, AWS Identity and Access Management (IAM) roles with permissions required during the preparation of the image, key pair, and all the parameters I usually select when I start an EC2 instance.

\n

\"Image

\n

Finally, I select where I want to distribute the output AMI. By default, it stays on my account. But I can also share or copy it to other accounts.

\n

\"Image

\n

Run the pipeline
Now I’m ready to run the pipeline. I select Image pipelines, then I select the pipeline I just created (Sonoma-Xcode). From the Actions menu, I select Run pipeline.

\n

\"Image

\n

I can observe the progress and the detailed logs from Amazon CloudWatch.

\n

After a while, the AMI is created and ready to use.

\n

\"Image

\n

Testing my AMI
To finish the demo, I start an EC2 Mac instance with the AMI I just created (remember to allocate a Dedicated Host first or to reuse the one you used for EC2 Image Builder).

\n

Once the instance is started, I connect to it using secure shell (SSH) and verify that Xcode is correctly installed.

\n

\"Image

\n

Pricing and availability
EC2 Image Builder for macOS is now available in all AWS Regions where EC2 Mac instances are available: US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Mumbai, Seoul, Singapore, Sydney, Tokyo), and Europe (Frankfurt, Ireland, London, Stockholm) (not all Mac instance types are available in all Regions).

\n

It comes at no additional cost, and you’re only charged for the resources in use during the pipeline execution, namely the time your EC2 Mac Dedicated Host is allocated, with a minimum of 24 hours.

\n

The preview of macOS support in EC2 Image Builder allows you to consolidate your image pipelines, automate your golden image creation processes, and use the benefits of cloud-focused integrations on AWS. As the EC2 Mac platform continues to expand with more instance types, this new capability positions EC2 Image Builder as a comprehensive solution for image management across Windows, Linux, and macOS.

\n

Create your first pipeline today! 

\n

-- seb

", "url": "https://aws.amazon.com/blogs/aws/ec2-image-builder-now-supports-building-and-testing-macos-images/", "title": "EC2 Image Builder now supports building and testing macOS images", "date_modified": "2024-10-23T17:52:19.000Z" }, { "content_html": "Comments", "url": "https://determinate.systems/posts/flakehub-cache-and-private-flakes/", "title": "Nix at work: FlakeHub Cache and private flakes", "date_modified": "2024-10-23T17:28:39.000Z" }, { "content_html": "Comments", "url": "https://blogsystem5.substack.com/p/bazelcon-2024-recap", "title": "BazelCon 2024 Recap", "date_modified": "2024-10-23T14:50:10.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/intro-access-for-infrastructure-ssh/", "title": "Fearless SSH: Short-lived certificates bring Zero Trust to infrastructure", "date_modified": "2024-10-23T09:44:36.000Z" }, { "content_html": "

Comments

", "url": "https://blogsystem5.substack.com/p/bazelcon-2024-recap", "title": "BazelCon 2024 recap", "date_modified": "2024-10-22T21:55:22.000Z" }, { "content_html": "

At the end of the day, developers build, test, deploy and maintain software. But like with lots of things, it’s about the journey, not the destination.

\n

Among platform engineers, we sometimes refer to that journey as the developer experience (DX), which encompasses how developers feel and interact with the tools and services they use throughout the software build, test, deployment and maintenance process.

\n

Prioritizing DX is essential: Frustrated developers lead to inefficiency and talent loss as well as to shadow IT. Conversely, a positive DX drives innovation, community, and productivity. And if you want to provide a  positive DX, you need to start measuring how you’re doing.

\n

At PlatformCon 2024, I gave a talk entitled \"Improving your developers' platform experience by applying Google frameworks and methods” where I spoke about Google’s HEART Framework, which provides a holistic view of your organization's developers’ experience through actionable data.

\n

In this article, I will share ideas on how you can apply the HEART framework to your Platform Engineering practice, to gain a more comprehensive view of your organization’s developer experience. But before I do that, let me explain what the HEART Framework is.

\n
\n
aside_block
\n
<ListValue: [StructValue([('title', 'Try Google Cloud for free'), ('body', <wagtail.rich_text.RichText object at 0x3e05a44c79a0>), ('btn_text', 'Get started for free'), ('href', 'https://console.cloud.google.com/freetrial?redirectPath=/welcome'), ('image', None)])]>
\n
\n

The HEART Framework: an introduction

\n

In a nutshell, HEART measures developer behaviors and attitudes from their experience of your platform and provides you with insights into what’s going on behind the numbers, by defining specific metrics to track progress towards goals. This is beneficial because continuous improvements through feedback are vital components of a platform engineering journey, helping both platform and application product teams make decisions that are data-driven and user-centered.

\n

However, HEART is not a data collection tool in and of itself; rather, it’s a user-sentiment framework for selecting the right metrics to focus on based on product or platform objectives. It balances quantitative or empirical data, e.g., number of active portal users, with qualitative or subjective insights such as \"My users feel the portal navigation is confusing.\" In other words, consider HEART as a framework or methodology for assessing user experience, rather than a specific tool or assessment. It helps you decide what to measure, not how to measure it.

\n
\n\n\n\n\n\n\n \n
\n
\n \n\n
\n\n \n \n \n \"image2\"\n \n \n \n
\n\n \n
\n
\n \n\n\n\n\n
\n

Let’s take a look at each of these in more detail.

\n

Happiness: Do users actually enjoy using your product?

\n

Highlight: Gathering and analyzing developer feedback

\n

Subjective metrics:

\n\n

System metrics:

\n\n

Watch out for: While platforms can boost developer productivity, they might not necessarily contribute to developer job satisfaction. This warrants further investigation, especially if your research suggests that your developers are unhappy.

\n

Engagement: What is the developer breadth and quality of platform experience?

\n

Highlight: Frequency of interaction between platform engineers with developers and quality of interaction — intensity and quality of interaction with the platform, participation on chat channels, training, dual ownership of golden paths, joint troubleshooting, engaging in architectural design discussions, and the breadth of interaction by everyone from new hires through to senior developers.

\n

Subjective metrics:

\n\n

System metrics:

\n\n

Watch out for: Don’t confuse engagement with satisfaction. Developers may rate the platform highly in surveys, but usage data might reveal low frequency of interaction with core features or a limited subset of teams actively using the platform. Ask them “How has the platform changed your daily workflow?” rather than \"Are you satisfied with the platform?”

\n

Adoption: What is the platform growth rate and developer feature adoption?

\n

Highlight: Overall acceptance and integration of the platform into the development workflow.

\n

System metrics:

\n\n

Watch out for: Overlooking the \"long tail\" of adoption. A platform might see a burst of early adoption, but then plateau or even decline if it fails to continuously evolve and meet changing developer needs. Don't just measure initial adoption, monitor how usage evolves over weeks, months, and years.

\n

Retention: Are developers loyal to the platform?

\n

Highlight: Long-term engagement and reducing churn.

\n

Subjective metrics:

\n\n

System metrics:

\n\n

Watch out for: Misinterpreting the reasons for churn. When developers stop using your platform (churn), it's crucial to understand why. Incorrectly identifying the cause can lead to wasted effort and missed opportunities for improvement. Consider factors outside the platform — churn could be caused by changes in project requirements, team structures or industry trends.

\n

Task success: Can developers complete specific tasks?

\n

Highlight: Efficiency and effectiveness of the platform in supporting specific developer activities.

\n

Subjective metrics:

\n\n

System metrics:

\n\n

Watch out for: Don't confuse task success with task completion. Simply measuring whether developers can complete tasks on the platform doesn't necessarily indicate true success. Developers might find workarounds or complete tasks inefficiently, even if they technically achieve the end goal. It may be worth manually observing developer workflows in their natural environment to identify pain points and areas of friction in their workflows.

\n

Also, be careful with misaligning task success with business goals. Task completion might overlook the broader impact on business objectives. A platform might enable developers to complete tasks efficiently, but if those tasks don't contribute to overall business goals, the platform's true value is questionable.

\n

Applying the HEART framework to platform engineering

\n

It’s not necessary to use all of the categories each time. The number of categories to consider really depends on the specific goals and context of the assessment; you can include everything or trim it down to better match your objective. Here are some examples:

\n\n

Keep in mind that relying on just one category will likely provide an incomplete picture.

\n

When should you use the framework?

\n

In a perfect world, you would use the HEART framework to establish a baseline assessment a few months after launching your platform, which will provide you with a valuable insight into early developer experience. As your platform evolves, this initial data becomes a benchmark for measuring progress and identifying trends. Early measurement allows you to proactively address UX issues, guide design decisions with data, and iterate quickly for optimal functionality and developer satisfaction. If you're starting with an MVP, conduct the baseline assessment once the core functionality is in place and you have a small group of early users to provide feedback.

\n

After 12 or more months of usage, you can also add metrics to embody a new or more mature platform. This can help you gather deeper insights into your developers’ experience by understanding how they are using the platform, measure the impact of changes you’ve made to the platform, or identify areas for improvement and prioritize future development efforts. If you've added new golden paths, tooling, or enhanced functionality, then you'll need to track metrics that measure their success and impact on developer behavior.

\n

The frequency with which you assess HEART metrics depends on several factors, including:

\n\n

Schedule periodic deep dives (e.g. quarterly or bi-annually) using the HEART framework to gain a more in-depth understanding of your platform's performance and identify areas for improvement.

\n

Taking more steps towards platform engineering

\n

In this blog post, we’ve shown how the HEART framework can be applied to platform engineering to measure and improve the developer experience. We’ve explored the five key aspects of the framework — happiness, engagement, adoption, retention, and task success — and provided specific metrics for each and guidance on when to apply them.By applying these insights, platform engineering teams can create a more positive and productive environment for their developers, leading to greater success in their software development efforts.To learn more about platform engineering, check out some of our other articles:  5 myths about platform engineering: what it is and what it isn’t, Another five myths about platform engineering, and Laying the foundation for a career in platform engineering.

\n

And finally, check out the DORA Report 2024, which now has a section on Platform Engineering.

", "url": "https://cloud.google.com/blog/products/application-development/how-platform-engineers-can-improve-their-developers-experience/", "title": "Measuring developer experience with the HEART Framework: A guide for platform engineers", "date_modified": "2024-10-22T17:00:00.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/announcing-planetscale-vectors-public-beta", "title": "The PlanetScale vectors public beta", "date_modified": "2024-10-22T16:06:55.000Z" }, { "content_html": "Comments", "url": "https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse", "title": "A new JSON data type for ClickHouse", "date_modified": "2024-10-22T14:47:00.000Z" }, { "content_html": "

Today, I am excited to announce the general availability of Amazon Aurora PostgreSQL-Compatible Edition and Amazon DynamoDB zero-ETL integrations with Amazon Redshift. Zero-ETL integration seamlessly makes transactional or operational data available in Amazon Redshift, removing the need to build and manage complex data pipelines that perform extract, transform, and load (ETL) operations. It automates the replication of source data to Amazon Redshift, simultaneously updating source data for you to use in Amazon Redshift for analytics and machine learning (ML) capabilities to derive timely insights and respond effectively to critical, time-sensitive events.

\n

Using these new zero-ETL integrations, you can run unified analytics on your data from different applications without having to build and manage different data pipelines to write data from multiple relational and non-relational data sources into a single data warehouse. In this post, I provide two step-by-step walkthroughs on how to get started with both Amazon Aurora PostgreSQL and Amazon DynamoDB zero-ETL integrations with Amazon Redshift.

\n

To create a zero-ETL integration, you specify a source and Amazon Redshift as the target. The integration replicates data from the source to the target data warehouse, making it available in Amazon Redshift seamlessly, and monitors the pipeline’s health.

\n

Let’s explore how these new integrations work. In this post, you will learn how to create zero-ETL integrations to replicate data from different source databases (Aurora PostgreSQL and DynamoDB) to the same Amazon Redshift cluster. You will also learn how to select multiple tables or databases from Aurora PostgreSQL source databases to replicate data to the same Amazon Redshift cluster. You will observe how zero-ETL integrations provide flexibility without the operational burden of building and managing multiple ETL pipelines.

\n

Getting started with Aurora PostgreSQL zero-ETL integration with Amazon Redshift
Before creating a database, I create a custom cluster parameter group because Aurora PostgreSQL zero-ETL integration with Amazon Redshift requires specific values for the Aurora DB cluster parameters. In the Amazon RDS console, I go to Parameter groups in the navigation pane. I choose Create parameter group.

\n

I enter custom-pg-aurora-postgres-zero-etl for Parameter group name and Description. I choose Aurora PostgreSQL for Engine type and aurora-postgresql16 for Parameter group family (zero-ETL integration works with PostgreSQL 16.4 or above versions). Finally, I choose DB Cluster Parameter Group for Type and choose Create.

\n

\n

Next, I edit the newly created cluster parameter group by choosing it on the Parameter groups page. I choose Actions and then choose Edit. I set the following cluster parameter settings:

\n \n

I choose Save Changes.

\n

\n

Next, I create an Aurora PostgreSQL database. When creating the database, you can set the configurations according to your needs. Remember to choose Aurora PostgreSQL (compatible with PostgreSQL 16.4 or above) from Available versions and the custom cluster parameter group (custom-pg-aurora-postgres-zero-etl in this case) for DB cluster parameter group in the Additional configuration section.

\n

After the database becomes available, I connect to the Aurora PostgreSQL cluster, create a database named books, create a table named book_catalog in the default schema for this database and insert sample data to use with zero-ETL integration.

\n

\n

To get started with zero-ETL integration, I use an existing Amazon Redshift data warehouse. To create and manage Amazon Redshift resources, visit the Amazon Redshift Getting Started Guide.

\n

In the Amazon RDS console, I go to the Zero-ETL integrations tab in the navigation pane and choose Create zero-ETL integration. I enter postgres-redshift-zero-etl for Integration identifier and Amazon Aurora zero-ETL integration with Amazon Redshift for Integration description. I choose Next.

\n

\n

On the next page, I choose Browse RDS databases to select the source database. For the Data filtering options, I use database.schema.table pattern. I include my table called book_catalog in Aurora PostgreSQL books database. The * in filters will replicate all book_catalog tables in all schemas within books database. I choose Include as filter type and enter books.*.book_catalog into the Filter expression field. I choose Next.

\n

\n

On the next page, I choose Browse Redshift data warehouses and select the existing Amazon Redshift data warehouse as the target. I must specify authorized principals and integration source on the target to enable Amazon Aurora to replicate into the data warehouse and enable case sensitivity. Amazon RDS can complete these steps for me during setup, or I can configure them manually in Amazon Redshift. For this demo, I choose Fix it for me and choose Next.

\n

\n

After the case sensitivity parameter and the resource policy for data warehouse are fixed, I choose Next on the next Add tags and encryption page. After I review the configuration, I choose Create zero-ETL integration.

\n

\n

After the integration succeeded, I choose the integration name to check the details.

\n

\n

Now, I need to create a database from integration to finish setting up. I go to the Amazon Redshift console, choose Zero-ETL integrations in the navigation pane and select the Aurora PostgreSQL integration I just created. I choose Create database from integration.

\n

\n

I choose books as Source named database and I enter zeroetl_aurorapg as the Destination database name. I choose Create database.

\n

\n

After the database is created, I return to the Aurora PostgreSQL integration page. On this page, I choose Query data to connect to the Amazon Redshift data warehouse to observe if the data is replicated. When I run a select query in the zeroetl_aurorapg database, I see that the data in book_catalog table is replicated to Amazon Redshift successfully.

\n

\n

As I said in the beginning, you can select multiple tables or databases from the Aurora PostgreSQL source database to replicate the data to the same Amazon Redshift cluster. To add another database to the same zero-ETL integration, all I have to do is to add another filter to the Data filtering options in the form of database.schema.table, replacing the database part with the database name I want to replicate. For this demo, I will select multiple tables to be replicated to the same data warehouse. I create another table named publisher in the Aurora PostgreSQL cluster and insert sample data to it.

\n

\n

I edit the Data filtering options to include publisher table for replication. To do this, I go to the postgres-redshift-zero-etl details page and choose Modify. I append books.*.publisher using comma in the Filter expression field. I choose Continue. I review the changes and choose Save changes. I observe that the Filtered data tables section on the integration details page has now 2 tables included for replication.

\n

\n

When I switch to the Amazon Redshift Query editor and refresh the tables, I can see that the new publisher table and its records are replicated to the data warehouse.

\n

\n

Now that I completed the Aurora PostgreSQL zero-ETL integration with Amazon Redshift, let’s create a DynamoDB zero-ETL integration with the same data warehouse.

\n

Getting started with DynamoDB zero-ETL integration with Amazon Redshift
In this part, I proceed to create an Amazon DynamoDB zero-ETL integration using an existing Amazon DynamoDB table named Book_Catalog. The table has 2 items in it:

\n

\n

I go to the Amazon Redshift console and choose Zero-ETL integrations in the navigation pane. Then, I choose the arrow next to the Create zero-ETL integration and choose Create DynamoDB integration. I enter dynamodb-redshift-zero-etl for Integration name and Amazon DynamoDB zero-ETL integration with Amazon Redshift for Description. I choose Next.

\n

\n

On the next page, I choose Browse DynamoDB tables and select the Book_Catalog table. I must specify a resource policy with authorized principals and integration sources, and enable point-in-time recovery (PITR) on the source table before I create an integration. Amazon DynamoDB can do it for me, or I can change the configuration manually. I choose Fix it for me to automatically apply the required resource policies for the integration and enable PITR on the DynamoDB table. I choose Next.

\n

\n

Then, I choose my existing Amazon Redshift Serverless data warehouse as the target and choose Next.

\n

\n

I choose Next again in the Add tags and encryption page and choose Create DynamoDB integration in the Review and create page.

\n

Now, I need to create a database from integration to finish setting up just like I did with Aurora PostgreSQL zero-ETL integration. In the Amazon Redshift console, I choose the DynamoDB integration and I choose Create database from integration. In the popup screen, I enter zeroetl_dynamodb as the Destination database name and choose Create database.

\n

\n

After the database is created, I go to the Amazon Redshift Zero-ETL integrations page and choose the DynamoDB integration I created. On this page, I choose Query data to connect to the Amazon Redshift data warehouse to observe if the data from DynamoDB Book_Catalog table is replicated. When I run a select query in the zeroetl_dynamodb database, I see that the data is replicated to Amazon Redshift successfully. Note that the data from DynamoDB is replicated in SUPER datatype column and can be accessed using PartiQL sql.

\n

\n

I insert another entry to the DynamoDB Book_Catalog table.

\n

\n

When I switch to the Amazon Redshift Query editor and refresh the select query, I can see that the new record is replicated to the data warehouse.

\n

\n

Zero-ETL integrations between Aurora PostgreSQL and DynamoDB with Amazon Redshift help you unify data from multiple database clusters and unlock insights in your data warehouse. Amazon Redshift allows cross-database queries and materialized views based off the multiple tables, giving you the opportunity to consolidate and simplify your analytics assets, improve operational efficiency, and optimize cost. You no longer have to worry about setting up and managing complex ETL pipelines.

\n

Now available
Aurora PostgreSQL zero-ETL integration with Amazon Redshift is now available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Hong Kong), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm) AWS Regions.

\n

Amazon DynamoDB zero-ETL integration with Amazon Redshift is now available in all commercial, China and GovCloud AWS Regions.

\n

For pricing information, visit the Amazon Aurora and Amazon DynamoDB pricing pages.

\n

To get started with this feature, visit Working with Aurora zero-ETL integrations with Amazon Redshift and Amazon Redshift Zero-ETL integrations documentation.

\n

— Esra

", "url": "https://aws.amazon.com/blogs/aws/amazon-aurora-postgresql-and-amazon-dynamodb-zero-etl-integrations-with-amazon-redshift-now-generally-available/", "title": "Amazon Aurora PostgreSQL and Amazon DynamoDB zero-ETL integrations with Amazon Redshift now generally available", "date_modified": "2024-10-15T19:59:43.000Z" }, { "content_html": "Comments", "url": "https://www.honeycomb.io/blog/engineers-checklist-logging-best-practices", "title": "Logging Best Practices: An Engineer's Checklist", "date_modified": "2024-10-15T14:15:21.000Z" }, { "content_html": "Comments", "url": "https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-storage-in-every-durable-object/", "title": "Zero-latency SQLite storage in every Durable Object", "date_modified": "2024-10-13T23:07:03.000Z" }, { "content_html": "Comments", "url": "https://longform.asmartbear.com/icp-ideal-customer-persona/", "title": "Why targeting an ICP brings 10x more customers than you expected", "date_modified": "2024-10-13T07:52:56.000Z" }, { "content_html": "

Comments

", "url": "https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar", "title": "The Road to OCIv2 Images: What's Wrong with Tar? (2019)", "date_modified": "2024-10-12T17:25:53.000Z" }, { "content_html": "

Comments

", "url": "https://curiosum.com/blog/packaging-elixir-application-with-nix", "title": "Packaging an Elixir/Phoenix application with Nix", "date_modified": "2024-10-11T19:54:26.000Z" }, { "content_html": "Comments", "url": "https://peq42.com/blog/gleam-a-basic-introduction/", "title": "Gleam: A Basic Introduction", "date_modified": "2024-10-11T18:57:33.000Z" }, { "content_html": "Comments", "url": "https://nixiesearch.substack.com/p/nixiesearch-running-lucene-over-s3", "title": "Nixiesearch: Running Lucene over S3, and why we're building a new search engine", "date_modified": "2024-10-10T09:11:26.000Z" }, { "content_html": "

Failure is an expected state in production systems, and no predictable failure of either software or hardware components should result in a negative experience for users. The exact failure mode may vary, but certain remediation steps must be taken after detection. A common example is when an error occurs on a server, rendering it unfit for production workloads, and requiring action to recover.

When operating at Cloudflare’s scale, it is important to ensure that our platform is able to recover from faults seamlessly. It can be tempting to rely on the expertise of world-class engineers to remediate these faults, but this would be manual, repetitive, unlikely to produce enduring value, and not scaling. In one word: toil; not a viable solution at our scale and rate of growth.

In this post we discuss how we built the foundations to enable a more scalable future, and what problems it has immediately allowed us to solve.

\n
\n

Growing pains

\n \n \n \n
\n

The Cloudflare Site Reliability Engineering (SRE) team builds and manages the platform that helps product teams deliver our extensive suite of offerings to customers. One important component of this platform is the collection of servers that power critical products such as Durable Objects, Workers, and DDoS mitigation. We also build and maintain foundational software services that power our product offerings, such as configuration management, provisioning, and IP address allocation systems.

As part of tactical operations work, we are often required to respond to failures in any of these components to minimize impact to users. Impact can vary from lack of access to a specific product feature, to total unavailability. The level of response required is determined by the priority, which is usually a reflection of the severity of impact on users. Lower-priority failures are more common — a server may run too hot, or experience an unrecoverable hardware error. Higher-priority failures are rare and are typically resolved via a well-defined incident response process, requiring collaboration with multiple other teams.

The commonality of lower-priority failures makes it obvious when the response required, as defined in runbooks, is “toilsome”. To reduce this toil, we had previously implemented a plethora of solutions to automate runbook actions such as manually-invoked shell scripts, cron jobs, and ad-hoc software services. These had grown organically over time and provided solutions on a case-by-case basis, which led to duplication of work, tight coupling, and lack of context awareness across the solutions.

We also care about how long it takes to resolve any potential impact on users. A resolution process which involves the manual invocation of a script relies on human action, increasing the Mean-Time-To-Resolve (MTTR) and leaving room for human error. This risks increasing the amount of errors we serve to users and degrading trust.

These problems proved that we needed a way to automatically heal these platform components. This especially applies to our servers, for which failure can cause impact across multiple product offerings. While we have mechanisms to automatically steer traffic away from these degraded servers, in some rare cases the breakage is sudden enough to be visible.

\n
\n

Solving the problem

\n \n \n \n
\n

To provide a more reliable platform, we needed a new component that provides a common ground for remediation efforts. This would remove duplication of work, provide unified context-awareness and increase development speed, which ultimately saves hours of engineering time and effort.

A good solution would not allow only the SRE team to auto-remediate, it would empower the entire company. The key to adding self-healing capability was a generic interface for all teams to self-service and quickly remediate failures at various levels: machine, service, network, or dependencies.

A good way to think about auto-remediation is in terms of workflows. A workflow is a sequence of steps to get to a desired outcome. This is not dissimilar to a manual shell script which executes what a human would otherwise do via runbook instructions. Because of this logical fit with workflows, we decided to adopt Temporal. 

Temporal is a durable execution platform which is useful to gracefully manage infrastructure failures such as network outages and transient failures in external service endpoints. This capability meant we only needed to build a way to schedule “workflow” tasks and have Temporal provide reliability guarantees. This allowed us to focus on building out the orchestration system to support the control and flow of workflow execution in our data centers. 

Temporal’s documentation provides a good introduction to writing Temporal workflows.

\n
\n

Building an Automatic Remediation System

\n \n \n \n
\n

Below, we describe how our automatic remediation system works. It is essentially a way to schedule tasks across our global network with built-in reliability guarantees. With this system, teams can serve their customers more reliably. An unexpected failure mode can be recognized and immediately mitigated, while the root cause can be determined later via a more detailed analysis.

\n
\n

Step one: we need a coordinator

\n \n \n \n
\n \n
\n \n

After our initial testing of Temporal, it was now possible to write workflows. But we needed a way to schedule workflow tasks from other internal services. The coordinator was built to serve this purpose, and became the primary mechanism for the authorisation and scheduling of workflows. 

The most important roles of the coordinator are authorisation, workflow task routing, and safety constraints enforcement. Each consumer is authorized via mTLS authentication, and the coordinator uses an ACL to determine whether to permit the execution of a workflow. An ACL configuration looks like the following example.

\n 
server_config {\n    enable_tls = true\n    [...]\n    route_rule {\n      name  = \"global_get\"\n      method = \"GET\"\n      route_patterns = [\"/*\"]\n      uris = [\"spiffe://example.com/worker-admin\"]\n    }\n    route_rule {\n      name = \"global_post\"\n      method = \"POST\"\n      route_patterns = [\"/*\"]\n      uris = [\"spiffe://example.com/worker-admin\"]\n      allow_public = true\n    }\n    route_rule {\n      name = \"public_access\"\n      method = \"GET\"\n      route_patterns = [\"/metrics\"]\n      uris = []\n      allow_public = true\n      skip_log_match = true\n    }\n}\n
\n

Each workflow specifies two key characteristics: where to run the tasks and the safety constraints, using an HCL configuration file. Example constraints could be whether to run on only a specific node type (such as a database), or if multiple parallel executions are allowed: if a task has been triggered too many times, that is a sign of a wider problem that might require human intervention. The coordinator uses the Temporal Visibility API to determine the current state of the executions in the Temporal cluster.

An example of a configuration file is shown below:

\n 
task_queue_target = \"<target>\"\n\n# The following entries will ensure that\n# 1. This workflow is not run at the same time in a 15m window.\n# 2. This workflow will not run more than once an hour.\n# 3. This workflow will not run more than 3 times in one day.\n#\nconstraint {\n    kind = \"concurency\"\n    value = \"1\"\n    period = \"15m\"\n}\n\nconstraint {\n    kind = \"maxExecution\"\n    value = \"1\"\n    period = \"1h\"\n}\n\nconstraint {\n    kind = \"maxExecution\"\n    value = \"3\"\n    period = \"24h\"\n    is_global = true\n}\n
\n \n
\n

Step two: Task Routing is amazing

\n \n \n \n
\n \n
\n \n

An unforeseen benefit of using a central Temporal cluster was the discovery of Task Routing. This feature allows us to schedule a Workflow/Activity on any server that has a running Temporal Worker, and further segment by the type of server, its location, etc. For this reason, we have three primary task queues — the general queue in which tasks can be executed by any worker in the datacenter, the node type queue in which tasks can only be executed by a specific node type in the datacenter, and the individual node queue where we target a specific node for task execution.

We rely on this heavily to ensure the speed and efficiency of automated remediation. Certain tasks can be run in datacenters with known low latency to an external resource, or a node type with better performance than others (due to differences in the underlying hardware). This reduces the amount of failure and latency we see overall in task executions. Sometimes we are also constrained by certain types of tasks that can only run on a certain node type, such as a database.

Task Routing also means that we can configure certain task queues to have a higher priority for execution, although this is not a feature we have needed so far. A drawback of task routing is that every Workflow/Activity needs to be registered to the target task queue, which is a common gotcha. Thankfully, it is possible to catch this failure condition with proper testing.

\n
\n

Step three: when/how to self-heal?

\n \n \n \n
\n

None of this would be relevant if we didn’t put it to good use. A primary design goal for the platform was to ensure we had easy, quick ways to trigger workflows on the most important failure conditions. The next step was to determine what the best sources to trigger the actions were. The answer to this was simple: we could trigger workflows from anywhere as long as they are properly authorized and detect the failure conditions accurately.

Example triggers are an alerting system, a log tailer, a health check daemon, or an authorized engineer via a chatbot. Such flexibility allows a high level of reuse, and permits to invest more in workflow quality and reliability.

As part of the solution, we built a daemon that is able to poll a signal source for any unwanted condition and trigger a configured workflow. We have initially found Prometheus useful as a source because it contains both service-level and hardware/system-level metrics. We are also exploring more event-based trigger mechanisms, which could eliminate the need to use precious system resources to poll for metrics.

We already had internal services that are able to detect widespread failure conditions for our customers, but were only able to page a human. With the adoption of auto-remediation, these systems are now able to react automatically. This ability to create an automatic feedback loop with our customers is the cornerstone of these self-healing capabilities and we continue to work on stronger signals, faster reaction times, and better prevention of future occurrences.

The most exciting part, however, is the future possibility. Every customer cares about any negative impact from Cloudflare. With this platform we can onboard several services (especially those that are foundational for the critical path) and ensure we react quickly to any failure conditions, even before there is any visible impact.

\n
\n

Step four: packaging and deployment

\n \n \n \n
\n

The whole system is written in golang, and a single binary can implement each role. We distribute it as an apt package or a container for maximum ease of deployment.

We deploy a Temporal-based worker to every server we intend to run tasks on, and a daemon in datacenters where we intend to automatically trigger workflows based on the local conditions. The coordinator is more nuanced since we rely on task routing and can trigger from a central coordinator, but we have also found value in running coordinators locally in the datacenters. This is especially useful in datacenters with less capacity or degraded performance, removing the need for a round-trip to schedule the workflows.

\n
\n

Step five: test, test, test

\n \n \n \n
\n

Temporal provides native mechanisms to test an entire workflow, via a comprehensive test suite that supports end-to-end, integration, and unit testing, which we used extensively to prevent regressions while developing. We also ensured proper test coverage for all the critical platform components, especially the coordinator.

Despite the ease of written tests, we quickly discovered that they were not enough. After writing workflows, engineers need an environment as close as possible to the target conditions. This is why we configured our staging environments to support quick and efficient testing. These environments receive the latest changes and point to a different (staging) Temporal cluster, which enables experimentation and easy validation of changes.

After a workflow is validated in the staging environment, we can then do a full release to production. It seems obvious, but catching simple configuration errors before releasing has saved us many hours in development/change-related-task time.

\n
\n

Deploying to production

\n \n \n \n
\n

As you can guess from the title of this post, we put this in production to automatically react to server-specific errors and unrecoverable failures. To this end, we have a set of services that are able to detect single-server failure conditions based on analyzed traffic data. After deployment, we have successfully mitigated potential impact by taking any errant single sources of failure out of production.

We have also created a set of workflows to reduce internal toil and improve efficiency. These workflows can automatically test pull requests on target machines, wipe and reset servers after experiments are concluded, and take away manual processes that cost many hours in toil.

Building a system that is maintained by several SRE teams has allowed us to iterate faster, and rapidly tackle long-standing problems. We have set ambitious goals regarding toil elimination and are on course to achieve them, which will allow us to scale faster by eliminating the human bottleneck.

\n
\n

Looking to the future

\n \n \n \n
\n

Our immediate plans are to leverage this system to provide a more reliable platform for our customers and drastically reduce operational toil, freeing up engineering resources to tackle larger-scale problems. We also intend to leverage more Temporal features such as Workflow Versioning, which will simplify the process of making changes to workflows by ensuring that triggered workflows run expected versions. 

 We are also interested in how others are solving problems using durable execution platforms such as Temporal, and general strategies to eliminate toil. If you would like to discuss this further, feel free to reach out on the Cloudflare Community and start a conversation!

If you’re interested in contributing to projects that help build a better Internet, our engineering teams are hiring.

", "url": "https://blog.cloudflare.com/improving-platform-resilience-at-cloudflare", "title": "Improving platform resilience at Cloudflare through automation", "date_modified": "2024-10-09T06:07:17.805Z" }, { "content_html": "Comments", "url": "https://www.signalfire.com/blog/devrel-for-startups", "title": "Unlocking the 'aha' moment: Developer relations for startups", "date_modified": "2024-10-07T16:40:30.000Z" }, { "content_html": "Comments", "url": "https://blog.drewolson.org/gleam-is-pragmatic/", "title": "Gleam Is Pragmatic", "date_modified": "2024-10-06T18:15:23.000Z" }, { "content_html": "Comments", "url": "https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6", "title": "Good Retry, Bad Retry: An Incident Story", "date_modified": "2024-10-06T08:56:08.000Z" }, { "content_html": "Comments", "url": "https://flawless.dev/docs/", "title": "Flawless is now in public beta", "date_modified": "2024-10-03T21:09:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/tqwewe/kameo", "title": "Show HN: Kameo – a Rust library for building fault-tolerant, async actors", "date_modified": "2024-10-02T18:22:10.000Z" }, { "content_html": "Comments", "url": "https://github.com/tqwewe/kameo", "title": "Show HN: Kameo – Fault-tolerant async actors built on Tokio", "date_modified": "2024-10-02T18:22:10.000Z" }, { "content_html": "Comments", "url": "https://threedots.tech/post/distributed-transactions-in-go", "title": "Distributed Transactions in Go: Read Before You Try", "date_modified": "2024-10-02T14:10:07.000Z" }, { "content_html": "Comments", "url": "https://threedots.tech/post/distributed-transactions-in-go", "title": "Distributed transactions in Go: Read before you try", "date_modified": "2024-10-02T14:10:07.000Z" }, { "content_html": "Comments", "url": "https://www.highlight.io/blog/alert-evaluations-incremental-merges-in-clickhouse", "title": "Alert Evaluations: Incremental Merges in ClickHouse", "date_modified": "2024-10-02T12:55:48.000Z" }, { "content_html": "Comments", "url": "https://automattic.com/2024/10/01/wpe-terms/", "title": "Automattic–WP Engine Term Sheet", "date_modified": "2024-10-02T08:41:46.000Z" }, { "content_html": "Comments", "url": "https://walzr.com/bop-spotter", "title": "Bop Spotter", "date_modified": "2024-09-30T06:09:53.000Z" }, { "content_html": "Comments", "url": "https://int10h.org/oldschool-pc-fonts/", "title": "The Ultimate Oldschool PC Font Pack", "date_modified": "2024-09-30T02:26:46.000Z" }, { "content_html": "Comments", "url": "https://ma.tt/2024/09/t3/", "title": "On With Theo / T3.gg", "date_modified": "2024-09-29T21:15:24.000Z" }, { "content_html": "Comments", "url": "https://jvns.ca/blog/2024/09/27/some-go-web-dev-notes/", "title": "Some Go web dev notes", "date_modified": "2024-09-29T14:36:23.000Z" }, { "content_html": "Comments", "url": "https://discord.com/blog/how-discord-stores-trillions-of-messages", "title": "How Discord stores trillions of messages", "date_modified": "2024-09-28T22:07:43.000Z" }, { "content_html": "Comments", "url": "https://wordpress.org/news/2024/09/wp-engine-reprieve/", "title": "WP Engine Reprieve", "date_modified": "2024-09-27T21:23:44.000Z" }, { "content_html": "Comments", "url": "https://devstarterpack.io/", "title": "Dev Starter Pack: The essential startup starter kit", "date_modified": "2024-09-27T13:06:02.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/container-platform-preview/", "title": "Our container platform is in production. It has GPUs. Here's an early look", "date_modified": "2024-09-27T13:05:53.000Z" }, { "content_html": "

I spent a lot of time in the past couple of weeks working on a website in Go\nthat may or may not ever see the light of day, but I learned a couple of things\nalong the way I wanted to write down. Here they are:

\n

go 1.22 now has better routing

\n

I’ve never felt motivated to learn any of the Go routing libraries\n(gorilla/mux, chi, etc), so I’ve been doing all my routing by hand, like this.

\n
\t// DELETE /records:\n\tcase r.Method == \"DELETE\" && n == 1 && p[0] == \"records\":\n\t\tif !requireLogin(username, r.URL.Path, r, w) {\n\t\t\treturn\n\t\t}\n\t\tdeleteAllRecords(ctx, username, rs, w, r)\n\t// POST /records/<ID>\n\tcase r.Method == \"POST\" && n == 2 && p[0] == \"records\" && len(p[1]) > 0:\n\t\tif !requireLogin(username, r.URL.Path, r, w) {\n\t\t\treturn\n\t\t}\n\t\tupdateRecord(ctx, username, p[1], rs, w, r)\n\n
\n

But apparently as of Go 1.22, Go\nnow has better support for routing in the standard library, so that code can be\nrewritten something like this:

\n
\tmux.HandleFunc(\"DELETE /records/\", app.deleteAllRecords)\n\tmux.HandleFunc(\"POST /records/{record_id}\", app.updateRecord)\n
\n

Though it would also need a login middleware, so maybe something more like\nthis, with a requireLogin middleware.

\n
\tmux.Handle(\"DELETE /records/\", requireLogin(http.HandlerFunc(app.deleteAllRecords)))\n
\n

a gotcha with the built-in router: redirects with trailing slashes

\n

One annoying gotcha I ran into was: if I make a route for /records/, then a\nrequest for /records will be redirected to /records/.

\n

I ran into an issue with this where sending a POST request to /records\nredirected to a GET request for /records/, which broke the POST request\nbecause it removed the request body. Thankfully Xe Iaso wrote a blog post about the exact same issue which made it\neasier to debug.

\n

I think the solution to this is just to use API endpoints like POST /records\ninstead of POST /records/, which seems like a more normal design anyway.

\n

sqlc automatically generates code for my db queries

\n

I got a little bit tired of writing so much boilerplate for my SQL queries, but\nI didn’t really feel like learning an ORM, because I know what SQL queries I\nwant to write, and I didn’t feel like learning the ORM’s conventions for\ntranslating things into SQL queries.

\n

But then I found sqlc, which will compile a query like this:

\n
\n-- name: GetVariant :one\nSELECT *\nFROM variants\nWHERE id = ?;\n\n
\n

into Go code like this:

\n
const getVariant = `-- name: GetVariant :one\nSELECT id, created_at, updated_at, disabled, product_name, variant_name\nFROM variants\nWHERE id = ?\n`\n\nfunc (q *Queries) GetVariant(ctx context.Context, id int64) (Variant, error) {\n\trow := q.db.QueryRowContext(ctx, getVariant, id)\n\tvar i Variant\n\terr := row.Scan(\n\t\t&i.ID,\n\t\t&i.CreatedAt,\n\t\t&i.UpdatedAt,\n\t\t&i.Disabled,\n\t\t&i.ProductName,\n\t\t&i.VariantName,\n\t)\n\treturn i, err\n}\n
\n

What I like about this is that if I’m ever unsure about what Go code to write\nfor a given SQL query, I can just write the query I want, read the generated\nfunction and it’ll tell me exactly what to do to call it. It feels much easier\nto me than trying to dig through the ORM’s documentation to figure out how to\nconstruct the SQL query I want.

\n

Reading Brandur’s sqlc notes from 2024 also gave me some confidence\nthat this is a workable path for my tiny programs. That post gives a really\nhelpful example of how to conditionally update fields in a table using CASE\nstatements (for example if you have a table with 20 columns and you only want\nto update 3 of them).

\n

sqlite tips

\n

Someone on Mastodon linked me to this post called Optimizing sqlite for servers. My projects are small and I’m\nnot so concerned about performance, but my main takeaways were:

\n\n

There are a more tips in that post that seem useful (like “COUNT queries are\nslow” and “Use STRICT tables”), but I haven’t done those yet.

\n

Also sometimes if I have two tables where I know I’ll never need to do a JOIN\nbeteween them, I’ll just put them in separate databases so that I can connect\nto them independently.

\n

Go 1.19 introduced a way to set a GC memory limit

\n

I run all of my Go projects in VMs with relatively little memory, like 256MB or\n512MB. I ran into an issue where my application kept getting OOM killed and it\nwas confusing – did I have a memory leak? What?

\n

After some Googling, I realized that maybe I didn’t have a memory leak, maybe I\njust needed to reconfigure the garbage collector! It turns out that by default (according to A Guide to the Go Garbage Collector), Go’s garbage collector will\nlet the application allocate memory up to 2x the current heap size.

\n

Mess With DNS’s base heap size is around 170MB and\nthe amount of memory free on the VM is around 160MB right now, so if its memory\ndoubled, it’ll get OOM killed.

\n

In Go 1.19, they added a way to tell Go “hey, if the application starts using\nthis much memory, run a GC”. So I set the GC memory limit to 250MB and it seems\nto have resulted in the application getting OOM killed less often:

\n
export GOMEMLIMIT=250MiB\n
\n

some reasons I like making websites in Go

\n

I’ve been making tiny websites (like the nginx playground) in Go on and off for the last 4 years or so and it’s really been working for me. I think I like it because:

\n\n

In general everything about it feels like it makes projects easy to work on for\n5 days, abandon for 2 years, and then get back into writing code without a lot\nof problems.

\n

For contrast, I’ve tried to learn Rails a couple of times and I really want\nto love Rails – I’ve made a couple of toy websites in Rails and it’s always\nfelt like a really magical experience. But ultimately when I come back to those\nprojects I can’t remember how anything works and I just end up giving up. It\nfeels easier to me to come back to my Go projects that are full of a lot of\nrepetitive boilerplate, because at least I can read the code and figure out how\nit works.

\n

things I haven’t figured out yet

\n

some things I haven’t done much of yet in Go:

\n\n

In general I’m not sure how to implement security-sensitive features so I don’t\nstart projects which need login/CSRF/etc. I imagine this is where a framework\nwould help.

\n

it’s cool to see the new features Go has been adding

\n

Both of the Go features I mentioned in this post (GOMEMLIMIT and the routing)\nare new in the last couple of years and I didn’t notice when they came out. It\nmakes me think I should pay closer attention to the release notes for new Go\nversions.

", "url": "https://jvns.ca/blog/2024/09/27/some-go-web-dev-notes/", "title": "Some Go web dev notes", "date_modified": "2024-09-27T11:16:00.000Z" }, { "content_html": "Comments", "url": "https://blog.trailofbits.com/2024/09/24/notes-on-aws-nitro-enclaves-attack-surface/", "title": "AWS Nitro Enclaves: Attack Surface", "date_modified": "2024-09-26T07:12:04.000Z" }, { "content_html": "Comments", "url": "https://www.artie.com/blogs/postgres-write-ahead-logs", "title": "Deep Dive into Postgres Write-Ahead Logs", "date_modified": "2024-09-24T23:32:36.000Z" }, { "content_html": "Comments", "url": "https://blog.oodle.ai/building-a-high-performance-low-cost-metrics-observability-system/", "title": "Show HN: Oodle – serverless, fully-managed, drop-in replacement for Prometheus", "date_modified": "2024-09-24T12:39:39.000Z" }, { "content_html": "

Comments

", "url": "https://xeiaso.net/notes/2024/small-nextjs-images/", "title": "Make your Next.JS Docker images microscopic", "date_modified": "2024-09-23T10:45:24.000Z" }, { "content_html": "Comments", "url": "https://laike9m.com/blog/no-one-builds-in-public,160/", "title": "Does “building in public” work?", "date_modified": "2024-09-22T17:30:55.000Z" }, { "content_html": "Comments", "url": "https://github.com/xataio/pgroll", "title": "pgroll: PostgreSQL zero-downtime migrations made easy", "date_modified": "2024-09-22T12:09:29.000Z" }, { "content_html": "Comments", "url": "https://wordpress.org/news/2024/09/wp-engine/", "title": "WP Engine is not WordPress", "date_modified": "2024-09-22T00:16:25.000Z" }, { "content_html": "Comments", "url": "https://www.rijksmuseum.nl/en/stories/operation-night-watch/story/ultra-high-resolution-image-of-the-night-watch", "title": "Ultra high-resolution image of The Night Watch", "date_modified": "2024-09-21T09:08:20.000Z" }, { "content_html": "Comments", "url": "https://discord.com/blog/how-discord-reduced-websocket-traffic-by-40-percent", "title": "Discord Reduced WebSocket Traffic by 40%", "date_modified": "2024-09-20T18:09:15.000Z" }, { "content_html": "Comments", "url": "https://www.inngest.com/", "title": "Show HN: Inngest 1.0 – Open-source durable workflows on every platform", "date_modified": "2024-09-20T17:33:15.000Z" }, { "content_html": "Comments", "url": "https://www.anthropic.com/news/contextual-retrieval", "title": "Anthropic – Introducing Contextual Retrieval", "date_modified": "2024-09-20T01:57:22.000Z" }, { "content_html": "Comments", "url": "http://blog.rongarret.info/2024/09/yes-you-can-have-exactly-once-delivery.html", "title": "Yes, you can have exactly-once delivery", "date_modified": "2024-09-20T01:33:17.000Z" }, { "content_html": "Comments", "url": "https://travel.stackexchange.com/questions/26780/is-there-a-flight-search-engine-that-combines-flights-from-different-airlines", "title": "Is there a flight search engine that combines flights from different airlines?", "date_modified": "2024-09-20T01:20:32.000Z" }, { "content_html": "Comments", "url": "https://dtinth.github.io/comic-mono-font/", "title": "Comic Mono", "date_modified": "2024-09-18T20:36:28.000Z" }, { "content_html": "Comments", "url": "https://www.swift.org/blog/announcing-swift-6/", "title": "Swift 6", "date_modified": "2024-09-17T19:20:58.000Z" }, { "content_html": "Comments", "url": "https://jeremymorrell.dev/blog/minimal-js-tracing/", "title": "OpenTelemetry Tracing in < 200 lines of code", "date_modified": "2024-09-17T17:21:41.000Z" }, { "content_html": "Comments", "url": "https://newsroom.ibm.com/blog-ibm-acquires-kubecost-to-broaden-hybrid-cloud-cost-management-capabilities", "title": "IBM acquires Kubecost", "date_modified": "2024-09-17T13:47:22.000Z" }, { "content_html": "Comments", "url": "https://jfmengels.net/constraints-and-guarantees/", "title": "Constraints and Guarantees", "date_modified": "2024-09-16T08:26:14.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/NickADobos/status/1835242067255238862", "title": "Mr Beast YouTube Playbook Leaked", "date_modified": "2024-09-16T00:06:04.000Z" }, { "content_html": "Comments", "url": "https://www.brailleinstitute.org/freefont/", "title": "Atkinson Hyperlegible Font", "date_modified": "2024-09-15T20:42:03.000Z" }, { "content_html": "Comments", "url": "https://delightyourusers.com/psychology", "title": "Product Psychology", "date_modified": "2024-09-13T13:22:17.000Z" }, { "content_html": "Comments", "url": "https://www.hadijaveed.me/2024/09/08/does-your-startup-really-need-complex-cloud-infrastructure/", "title": "Does your startup need complex cloud infrastructure?", "date_modified": "2024-09-13T02:29:51.000Z" }, { "content_html": "Comments", "url": "https://dashbit.co/blog/remix-concurrent-submissions-flawed", "title": "Remix's concurrent submissions are fundamentally flawed", "date_modified": "2024-09-12T12:47:31.000Z" }, { "content_html": "Comments", "url": "https://briefer.cloud/blog/posts/open-source-strategy/", "title": "Going open-source as a VC-Backed company", "date_modified": "2024-09-10T13:16:28.000Z" }, { "content_html": "Comments", "url": "https://blog.gitbutler.com/why-github-actually-won/", "title": "Why GitHub Won", "date_modified": "2024-09-09T16:27:29.000Z" }, { "content_html": "Comments", "url": "https://blog.gitbutler.com/why-github-actually-won/", "title": "Why GitHub won", "date_modified": "2024-09-09T16:27:29.000Z" }, { "content_html": "Comments", "url": "https://github.com/keisku/gmon", "title": "Show HN: Goroutine Monitor Powered by eBPF", "date_modified": "2024-09-08T10:19:46.000Z" }, { "content_html": "Comments", "url": "https://github.com/cloudflare/serverless-registry", "title": "Serverless-registry: A Docker registry backed by Workers and R2", "date_modified": "2024-09-05T16:34:51.000Z" }, { "content_html": "Comments", "url": "https://ferd.ca/a-pipeline-made-of-airbags.html", "title": "A Pipeline Made of Airbags", "date_modified": "2024-09-05T14:11:48.000Z" }, { "content_html": "Comments", "url": "https://pentacent.com/blog/leaving-the-cloud-broken-paas-promise/", "title": "Cloud of Disillusion: The Broken Promise of PaaS", "date_modified": "2024-09-04T15:26:39.000Z" }, { "content_html": "Comments", "url": "https://hynek.me/articles/docker-uv/", "title": "Production-ready Docker Containers with uv", "date_modified": "2024-09-04T13:27:46.000Z" }, { "content_html": "

Comments

", "url": "https://departuremono.com/", "title": "Departure Mono", "date_modified": "2024-09-02T17:41:06.000Z" }, { "content_html": "Comments", "url": "https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/", "title": "Notes on Distributed Systems for Young Bloods", "date_modified": "2024-09-02T17:23:26.000Z" }, { "content_html": "Comments", "url": "https://github.com/Glimesh/broadcast-box", "title": "Show HN: OBS Live-streaming with 120ms latency", "date_modified": "2024-09-02T12:46:47.000Z" }, { "content_html": "Comments", "url": "https://orbstack.dev/", "title": "OrbStack: The fast, light, and easy way to run Docker containers and Linux", "date_modified": "2024-09-02T01:35:56.000Z" }, { "content_html": "Comments", "url": "https://kittenlabs.de/blog/2024/09/01/extreme-pi-boot-optimization/", "title": "Extreme Pi Boot Optimization", "date_modified": "2024-09-01T21:36:55.000Z" }, { "content_html": "Comments", "url": "https://rockyj-blogs.web.app/2024/08/31/shine-with-gleam.html", "title": "Shine with Gleam", "date_modified": "2024-08-31T08:42:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/newsnowlabs/runcvm", "title": "RunCVM: An open-source Docker runtime for launching container images in VMs", "date_modified": "2024-08-28T22:38:13.000Z" }, { "content_html": "Comments", "url": "https://owickstrom.github.io/the-monospace-web/", "title": "The Monospace Web", "date_modified": "2024-08-27T17:09:56.000Z" }, { "content_html": "Comments", "url": "https://www.bly.com/Pages/documents/STIKFS.html", "title": "Marketing to Engineers (2001)", "date_modified": "2024-08-27T15:23:04.000Z" }, { "content_html": "Comments", "url": "https://www.morling.dev/blog/leader-election-with-s3-conditional-writes/", "title": "Leader Election with S3 Conditional Writes", "date_modified": "2024-08-26T13:54:08.000Z" }, { "content_html": "Comments", "url": "https://blog.colinbreck.com/predicting-the-future-of-distributed-systems/", "title": "Predicting the future of distributed systems", "date_modified": "2024-08-26T11:56:19.000Z" }, { "content_html": "Comments", "url": "https://blog.api-fiddle.com/posts/coolify-revolution", "title": "Coolify’s rise to fame, and why it could be a big deal", "date_modified": "2024-08-26T11:50:20.000Z" }, { "content_html": "Comments", "url": "https://www.ubicloud.com/blog/building-infrastructure-control-planes-in-ruby", "title": "13 Years of Building Infrastructure Control Planes in Ruby", "date_modified": "2024-08-26T10:48:50.000Z" }, { "content_html": "Comments", "url": "https://github.com/moxie0/knockknock", "title": "Knockknock: Simple, secure, and stealthy port knocking implementation", "date_modified": "2024-08-24T11:49:53.000Z" }, { "content_html": "Comments", "url": "https://github.com/beac0n/ruroco", "title": "Show HN: Ruroco – like port knocking, but better", "date_modified": "2024-08-23T10:19:07.000Z" }, { "content_html": "Comments", "url": "https://playinzoi.com/en", "title": "PUBG developer Krafton make a life simulation game inzoi", "date_modified": "2024-08-21T01:10:29.000Z" }, { "content_html": "Comments", "url": "https://www.ycombinator.com/launches/Ler-ares-industries-building-low-cost-cruise-missiles", "title": "Launch YC: Ares Industries – Building low-cost cruise missiles", "date_modified": "2024-08-21T01:04:06.000Z" }, { "content_html": "Comments", "url": "https://www.zen-browser.app/", "title": "Zen, a Arc-like open-source browser based on the Firefox engine", "date_modified": "2024-08-20T20:57:04.000Z" }, { "content_html": "

Comments

", "url": "https://www.youtube.com/playlist?list=PL9eL-xg48OM3aoODepNLa2Qqq67CDsSKf", "title": "Systems Distributed '24", "date_modified": "2024-08-17T08:33:02.000Z" }, { "content_html": "Comments", "url": "https://blackmesa.com/", "title": "Black Mesa", "date_modified": "2024-08-15T22:07:25.000Z" }, { "content_html": "Comments", "url": "https://github.com/project-oak/oak", "title": "Project Oak: Meaningful control of data in distributed systems", "date_modified": "2024-08-14T14:00:55.000Z" }, { "content_html": "Comments", "url": "https://www.blacksmith.sh/blog/faster-docker-builds-using-a-remote-buildkit-instance", "title": "Faster Docker builds using a remote BuildKit instance", "date_modified": "2024-08-13T00:11:04.000Z" }, { "content_html": "

Comments

", "url": "https://kokada.capivaras.dev/blog/building-static-binaries-in-nix/", "title": "Building static binaries in Nix", "date_modified": "2024-08-11T19:35:54.000Z" }, { "content_html": "Comments", "url": "https://servermono.com/", "title": "Server Mono: A Typeface Inspired by Typewriters, Apple's SF Mono, and CLIs", "date_modified": "2024-08-11T16:04:49.000Z" }, { "content_html": "Comments", "url": "https://www.helpnetsecurity.com/2024/08/07/github-bitbucket-gitlab-jira-incidents/", "title": "Number of incidents affecting GitHub, Bitbucket, Gitlab and Jira is rising", "date_modified": "2024-08-09T08:48:50.000Z" }, { "content_html": "Comments", "url": "https://slimsaas.com/blog/saas-copy-writing-saas-marketing-framework-works", "title": "SaaS Copywriting: Marketing SaaS Framework", "date_modified": "2024-08-09T01:49:22.000Z" }, { "content_html": "Comments", "url": "https://www.enterpriseintegrationpatterns.com/ramblings/queues_flow_control.html", "title": "Queues invert control flow but require flow control", "date_modified": "2024-08-08T20:22:52.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/introducing-automatic-ssl-tls-securing-and-simplifying-origin-connectivity", "title": "Cloudflare Introduces Automatic SSL/TLS", "date_modified": "2024-08-08T16:44:18.000Z" }, { "content_html": "Comments", "url": "https://jepsen.io/analyses/jetcd-0.8.2", "title": "Jepsen: Jetcd 0.8.2", "date_modified": "2024-08-08T14:09:14.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=41182055", "title": "Launch HN: Release (YC W20) – Orchestrate AI Infrastructure and Applications", "date_modified": "2024-08-07T14:50:11.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=41180492", "title": "Ask HN: How to Price a Product", "date_modified": "2024-08-07T12:02:27.000Z" }, { "content_html": "Comments", "url": "https://news.alvaroduran.com/p/cringey-but-true-how-uber-tests-payments", "title": "Cringey, but True: How Uber Tests Payments in Production", "date_modified": "2024-08-07T07:16:02.000Z" }, { "content_html": "

Comments

", "url": "https://ntietz.com/blog/first-impressions-of-gleam", "title": "First impressions of Gleam: lots of joys and some rough edges", "date_modified": "2024-08-07T00:00:31.000Z" }, { "content_html": "Comments", "url": "https://lsd.gnunet.org/lsd0004/draft-schanzen-r5n.html", "title": "R5N - Obfuscated mesh routing on hostile networks.", "date_modified": "2024-08-03T04:30:27.000Z" }, { "content_html": "Comments", "url": "https://techcrunch.com/2024/07/30/real-time-database-startup-clickhouse-acquires-peerdb-to-expand-its-postgres-support/", "title": "ClickHouse acquires PeerDB to expand its Postgres support", "date_modified": "2024-08-02T15:57:33.000Z" }, { "content_html": "Comments", "url": "https://blog.trailofbits.com/2024/07/30/our-audit-of-homebrew/", "title": "Our Audit of Homebrew", "date_modified": "2024-07-30T22:39:21.000Z" }, { "content_html": "Comments", "url": "https://trayce.dev", "title": "Show HN: Trayce – Network tab for your local Docker containers", "date_modified": "2024-07-30T18:58:05.000Z" }, { "content_html": "Comments", "url": "https://blog.healthchecks.io/2024/07/running-one-man-saas-9-years-in/", "title": "Running One-man SaaS for 9 Years", "date_modified": "2024-07-29T22:15:53.000Z" }, { "content_html": "Comments", "url": "https://blog.healthchecks.io/2024/07/running-one-man-saas-9-years-in/", "title": "One-man SaaS, 9 Years In", "date_modified": "2024-07-29T22:15:53.000Z" }, { "content_html": "Comments", "url": "https://trayce.dev", "title": "Show HN: Trayce – Network tab for Docker containers", "date_modified": "2024-07-29T19:23:57.000Z" }, { "content_html": "Comments", "url": "http://jpetazzo.github.io/2024/07/26/cloudflare-images-overcharge-billing/", "title": "Is Cloudflare overcharging us for their images service?", "date_modified": "2024-07-29T14:55:57.000Z" }, { "content_html": "Comments", "url": "https://blog.val.town/blog/node-spawn-performance/", "title": "Why is spawning a new process in Node so slow?", "date_modified": "2024-07-26T23:07:24.000Z" }, { "content_html": "Comments", "url": "https://blog.thinkst.com/2024/07/unfashionably-secure-why-we-use-isolated-vms.html", "title": "Unfashionably secure: why we use isolated VMs", "date_modified": "2024-07-25T17:00:03.000Z" }, { "content_html": "Comments", "url": "https://www.joelonsoftware.com/2001/04/21/dont-let-architecture-astronauts-scare-you/", "title": "Don't Let Architecture Astronauts Scare You", "date_modified": "2024-07-25T13:30:22.000Z" }, { "content_html": "Comments", "url": "https://blog.replay.io/a-new-direction", "title": "Replay.io is discontinuing Replay Test Suites", "date_modified": "2024-07-25T08:19:57.000Z" }, { "content_html": "Comments", "url": "https://thereader.mitpress.mit.edu/the-rich-history-of-ham-radio-culture/", "title": "The Rich History of Ham Radio Culture", "date_modified": "2024-07-24T19:05:11.000Z" }, { "content_html": "

Comments

", "url": "https://blog.appsignal.com/2024/07/23/enhancing-your-elixir-codebase-with-gleam.html", "title": "Enhancing Your Elixir Codebase with Gleam", "date_modified": "2024-07-23T23:27:32.000Z" }, { "content_html": "Comments", "url": "https://syscall.sh/", "title": "Syscall.sh", "date_modified": "2024-07-21T07:13:39.000Z" }, { "content_html": "

Comments

", "url": "https://www.vincentschmalbach.com/google-now-defaults-to-not-indexing-your-content/", "title": "Google Now Defaults to Not Indexing Your Content", "date_modified": "2024-07-17T20:04:53.000Z" }, { "content_html": "Comments", "url": "https://www.reuters.com/markets/deals/google-backed-software-developer-gitlab-explores-sale-sources-say-2024-07-17/", "title": "Gitlab Explores Sale", "date_modified": "2024-07-17T07:57:26.000Z" }, { "content_html": "Comments", "url": "https://pippy.dev/", "title": "Show HN: Pippy – Pipelines for GitHub Actions", "date_modified": "2024-07-17T07:17:23.000Z" }, { "content_html": "

Comments

", "url": "https://xata.io/blog/multi-version-schema-migrations", "title": "Introducing multi-version schema migrations for Postgres", "date_modified": "2024-07-16T17:01:51.000Z" }, { "content_html": "Comments", "url": "https://blog.izissise.net/posts/uki/", "title": "Create Unified Kernel Image from Scratch", "date_modified": "2024-07-15T17:59:08.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/Articles/978738/", "title": "Rust for Filesystems", "date_modified": "2024-07-15T09:39:17.000Z" }, { "content_html": "Comments", "url": "https://www.checklyhq.com/blog/300ms-from-every-pod-startup-with-a-single-grafana-query/", "title": "We saved $5k a month with a single Grafana query", "date_modified": "2024-07-12T11:57:20.000Z" }, { "content_html": "Comments", "url": "https://ochagavia.nl/blog/using-s3-as-a-container-registry/", "title": "Using S3 as a Container Registry", "date_modified": "2024-07-12T04:26:23.000Z" }, { "content_html": "Comments", "url": "https://github.com/aws/aws-secretsmanager-agent", "title": "AWS Secrets Manager Agent", "date_modified": "2024-07-11T23:09:01.000Z" }, { "content_html": "Comments", "url": "https://xata.io/blog/postgres-free-tier", "title": "The economics of a Postgres free tier", "date_modified": "2024-07-11T17:43:58.000Z" }, { "content_html": "Comments", "url": "https://github.com/gojue/ecapture", "title": "Capturing Linux SSL/TLS plaintext without a CA certificate using eBPF", "date_modified": "2024-07-11T17:31:48.000Z" }, { "content_html": "Comments", "url": "https://mydiplomaticlife.com/how-to-survive-3-years-in-north-korea-as-a-foreigner/", "title": "How to Survive 3 Years in North Korea as a Foreigner", "date_modified": "2024-07-11T15:52:14.000Z" }, { "content_html": "Comments", "url": "https://typesetinthefuture.com/2018/12/04/walle/", "title": "The Typeset of Wall·E", "date_modified": "2024-07-11T09:28:54.000Z" }, { "content_html": "Comments", "url": "https://support.stripe.com/questions/july-2024-changes-to-stripe-billing", "title": "Changes to Stripe Billing", "date_modified": "2024-07-10T21:07:25.000Z" }, { "content_html": "Comments", "url": "https://turbopuffer.com/blog/turbopuffer", "title": "Turbopuffer: Fast search on object storage", "date_modified": "2024-07-09T14:48:59.000Z" }, { "content_html": "Comments", "url": "https://gleam.run/news/auto-imports-and-tolerant-expressions/", "title": "Gleam v1.3.0 – Auto-imports and tolerant expressions", "date_modified": "2024-07-09T13:35:25.000Z" }, { "content_html": "Comments", "url": "https://rye.astral.sh/", "title": "Rye: A Hassle-Free Python Experience", "date_modified": "2024-07-09T01:18:15.000Z" }, { "content_html": "Comments", "url": "https://trigger.dev/blog/event-loop-lag", "title": "How we tamed Node.js event loop lag: a deepdive", "date_modified": "2024-07-08T20:52:01.000Z" }, { "content_html": "Comments", "url": "https://wippler.dev/posts/synchronization-is-bad-for-scale", "title": "Synchronization Is Bad for Scale", "date_modified": "2024-07-07T17:03:16.000Z" }, { "content_html": "Comments", "url": "https://thenewstack.io/devops-isnt-dead-but-its-not-in-great-health-either/", "title": "DevOps Isn't Dead, but It's Not in Great Health Either", "date_modified": "2024-07-06T06:22:03.000Z" }, { "content_html": "Comments", "url": "https://code.visualstudio.com/docs/devcontainers/containers", "title": "Developing Inside a Container", "date_modified": "2024-07-03T10:09:05.000Z" }, { "content_html": "Comments", "url": "https://nextword.substack.com/p/why-ai-infrastructure-startups-are", "title": "Why AI Infrastructure Startups Are Insanely Hard to Build", "date_modified": "2024-07-03T03:15:58.000Z" }, { "content_html": "Comments", "url": "https://www.propelauth.com/post/nextjs-challenges", "title": "It's not just you, Next.js is getting harder to use", "date_modified": "2024-06-29T08:00:56.000Z" }, { "content_html": "Comments", "url": "https://logical.li/blog/devops/", "title": "DevOps: The Funeral", "date_modified": "2024-06-29T00:54:22.000Z" }, { "content_html": "Comments", "url": "https://matduggan.com/a-eulogy-for-devops/", "title": "A Eulogy for DevOps", "date_modified": "2024-06-28T22:59:05.000Z" }, { "content_html": "

Comments

", "url": "https://determinate.systems/posts/parallel-nix-eval", "title": "Adding parallel evaluation to Nix", "date_modified": "2024-06-27T15:19:16.000Z" }, { "content_html": "Comments", "url": "https://kmcd.dev/posts/grpc-the-bad-parts/", "title": "gRPC: The Bad Parts", "date_modified": "2024-06-26T11:30:04.000Z" }, { "content_html": "Comments", "url": "https://multi.app/blog/multi-is-joining-openai", "title": "OpenAI Acquires Multi", "date_modified": "2024-06-24T15:32:30.000Z" }, { "content_html": "Comments", "url": "https://criu.org/Main_Page", "title": "CRIU, a project to implement checkpoint/restore functionality for Linux", "date_modified": "2024-06-21T16:59:52.000Z" }, { "content_html": "Comments", "url": "https://www.cloudflarestatus.com/incidents/p7l6rrbhysck", "title": "Cloudflare Connectivity Issues in Eastern US and Central Europe", "date_modified": "2024-06-20T18:31:33.000Z" }, { "content_html": "Comments", "url": "https://www.dbos.dev/blog/dbos-vs-aws-step-functions-benchmark", "title": "Making Serverless Orchestration 25x Faster", "date_modified": "2024-06-17T18:24:25.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40699462", "title": "8 days downtime: Cloudflare r2 subscription bug ruins my business", "date_modified": "2024-06-16T19:30:58.000Z" }, { "content_html": "Comments", "url": "https://discourse.nixos.org/t/brew-nix-a-flake-automatically-packaging-all-homebrew-casks/44880", "title": "Brew-Nix: a flake automatically packaging all homebrew casks", "date_modified": "2024-06-14T02:16:46.000Z" }, { "content_html": "Comments", "url": "https://restate.dev/", "title": "Show HN: Restate – Low-latency durable workflows for JavaScript/Java, in Rust", "date_modified": "2024-06-12T15:25:23.000Z" }, { "content_html": "Comments", "url": "https://elixir-lang.org/blog/2024/06/12/elixir-v1-17-0-released/", "title": "Elixir 1.17 released: set-theoretic types in patterns, durations, OTP 27", "date_modified": "2024-06-12T11:13:52.000Z" }, { "content_html": "Comments", "url": "https://londonist.substack.com/p/a-new-map-of-medieval-london", "title": "A new map of medieval London", "date_modified": "2024-06-12T07:54:37.000Z" }, { "content_html": "Comments", "url": "https://christopher.engineering/en/blog/gleam-overview/", "title": "Exploring Gleam, a type-safe language on the BEAM", "date_modified": "2024-06-11T06:29:49.000Z" }, { "content_html": "

Comments

", "url": "https://codesandbox.io/blog/how-we-scale-our-microvm-infrastructure-using-low-latency-memory-decompression", "title": "microVM infrastructure using low-latency memory decompression", "date_modified": "2024-06-08T00:09:15.000Z" }, { "content_html": "Comments", "url": "https://github.com/ory/kratos/releases/tag/v1.2.0", "title": "Show HN: OSS Auth0 Alternative Ory Kratos Now with Full PassKey Support", "date_modified": "2024-06-06T12:38:58.000Z" }, { "content_html": "Comments", "url": "https://www.dropbox.com/scl/fi/6i1800ml23i6x4wn6jp0q/Enron-Summer.pdf?rlkey=i6eyhsa658w4la0vzkdhfy7uj&e=1&st=2yjulw6q&dl=0", "title": "Memories of an Enron Summer", "date_modified": "2024-06-05T21:01:57.000Z" }, { "content_html": "Comments", "url": "https://pigweed.dev/docs/blog/02-bazel-feature-flags.html", "title": "Feature flags in Bazel builds", "date_modified": "2024-06-05T18:26:02.000Z" }, { "content_html": "Comments", "url": "https://coder.com/blog/delivering-5x-faster-throughput-in-coder-2-12-0", "title": "We Improved the Performance of a Userspace TCP Stack in Go by 5X", "date_modified": "2024-06-05T16:13:39.000Z" }, { "content_html": "Comments", "url": "https://coder.com/blog/delivering-5x-faster-throughput-in-coder-2-12-0", "title": "We improved the performance of a userspace TCP stack in Go", "date_modified": "2024-06-05T16:13:39.000Z" }, { "content_html": "Comments", "url": "https://chrisdown.name/2018/01/02/in-defence-of-swap.html", "title": "In defence of swap: common misconceptions", "date_modified": "2024-06-05T06:42:42.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/", "title": "GitHub now provides Arm-based runners", "date_modified": "2024-06-05T05:02:09.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/", "title": "Arm64 on GitHub Actions", "date_modified": "2024-06-05T05:02:09.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40573852", "title": "Ask HN: How to bring traffic to my product", "date_modified": "2024-06-04T12:34:51.000Z" }, { "content_html": "Comments", "url": "https://brioche.dev/blog/announcing-brioche/", "title": "Show HN: Brioche – A new Nix-like package manager", "date_modified": "2024-06-03T16:00:04.000Z" }, { "content_html": "Comments", "url": "https://duckdb.org/2024/06/03/announcing-duckdb-100.html", "title": "DuckDB 1.0.0", "date_modified": "2024-06-03T13:18:18.000Z" }, { "content_html": "Comments", "url": "https://github.com/andyk/ht", "title": "ht: Headless Terminal", "date_modified": "2024-06-02T07:53:14.000Z" }, { "content_html": "Comments", "url": "https://bbozzay.com/posts/dev/multi-tenant-auth-zenstack", "title": "Multi-Tenant Authorization with Zenstack", "date_modified": "2024-06-01T12:19:06.000Z" }, { "content_html": "Comments", "url": "https://www.lux.camera/orion-from-idea-to-launch-in-45-days/", "title": "Orion – From idea to launch in 45 days", "date_modified": "2024-05-31T08:03:04.000Z" }, { "content_html": "Comments", "url": "https://andrewkelley.me/post/zig-new-cli-progress-bar-explained.html", "title": "Zig's new CLI progress bar explained", "date_modified": "2024-05-30T04:07:11.000Z" }, { "content_html": "Comments", "url": "https://github.com/caddyserver/caddy/releases/tag/v2.8.0", "title": "Caddy 2.8", "date_modified": "2024-05-29T22:04:24.000Z" }, { "content_html": "Comments", "url": "https://www.lux.camera/introducing-kino-pro-video-camera/", "title": "Kino: Pro Video Camera", "date_modified": "2024-05-29T17:06:40.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40510627", "title": "Ask HN: Can a website kill my internet connection? (WebRTC)", "date_modified": "2024-05-29T11:09:14.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=8zj7ei5Egk8", "title": "Cloudflare has become a pig butchering scam but legal [video]", "date_modified": "2024-05-29T08:45:36.000Z" }, { "content_html": "Comments", "url": "https://cedardb.com/blog/ode_to_postgres/", "title": "CedarDB: German-Powered, PostgreSQL-Compatiable Freak of Nature Database System", "date_modified": "2024-05-28T16:07:06.000Z" }, { "content_html": "Comments", "url": "https://www.ubicloud.com/blog/how-we-enabled-arm64-vms", "title": "How we enabled ARM64 VMs", "date_modified": "2024-05-27T15:02:19.000Z" }, { "content_html": "Comments", "url": "https://www.cmtops.dev/posts/rootless-docker-in-multiuser-environment/", "title": "Rootless Docker in a multi-user environment", "date_modified": "2024-05-25T19:11:40.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40456959", "title": "Show HN: Porter Cloud – PaaS with an eject button", "date_modified": "2024-05-23T16:47:00.000Z" }, { "content_html": "Comments", "url": "https://depot.dev/blog/faster-ec2-boot-time", "title": "Making EC2 boot time faster", "date_modified": "2024-05-23T14:31:36.000Z" }, { "content_html": "Comments", "url": "https://magicui.design/", "title": "Magic UI: UI Library for Design Engineers", "date_modified": "2024-05-23T03:23:54.000Z" }, { "content_html": "Comments", "url": "https://materializedview.io/p/s3-is-showing-its-age", "title": "S3 is showing its age", "date_modified": "2024-05-22T18:23:17.000Z" }, { "content_html": "

Comments

", "url": "https://blog.nelhage.com/post/stripe-dev-environment/", "title": "Stripe's monorepo developer environment", "date_modified": "2024-05-21T19:42:33.000Z" }, { "content_html": "

Comments

", "url": "https://git.sr.ht/~alip/syd/tree/main/item/doc/toctou-or-gtfo.md", "title": "State of Sandboxing in Linux", "date_modified": "2024-05-21T16:16:50.000Z" }, { "content_html": "Comments", "url": "https://www.backblaze.com/blog/how-backblaze-scales-our-storage-cloud/", "title": "Backblaze Scales Storage Cloud", "date_modified": "2024-05-21T16:14:06.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40418885", "title": "Ask HN: Why do you all think that Htmx is such a recent development?", "date_modified": "2024-05-20T19:00:12.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40408515", "title": "Ask HN: Video streaming is expensive yet YouTube \"seems\" to do it for free. How?", "date_modified": "2024-05-19T17:51:41.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40376857", "title": "Ask HN: SaaS Subscription or Usage-Based Pricing?", "date_modified": "2024-05-16T10:35:45.000Z" }, { "content_html": "Comments", "url": "https://jepsen.io/analyses/datomic-pro-1.0.7075", "title": "Jepsen: Datomic Pro 1.0.7075", "date_modified": "2024-05-15T16:57:30.000Z" }, { "content_html": "Comments", "url": "https://www.threads.net/@mikeyk/post/C6_feeSr0Q4", "title": "Mike Krieger: I've joined AnthropicAI as their Chief Product Officer", "date_modified": "2024-05-15T16:03:24.000Z" }, { "content_html": "Comments", "url": "https://github.com/quarylabs/quary", "title": "Show HN: Open-source BI and analytics for engineers", "date_modified": "2024-05-15T14:02:35.000Z" }, { "content_html": "Comments", "url": "https://blog.jak-linux.org/2024/05/14/solver3/", "title": "The new APT 3.0 solver", "date_modified": "2024-05-14T18:40:50.000Z" }, { "content_html": "Comments", "url": "https://www.highlight.io/blog/lw5-clickhouse-performance-optimization", "title": "Optimizing ClickHouse: Tactics that worked for us", "date_modified": "2024-05-14T14:57:55.000Z" }, { "content_html": "Comments", "url": "https://dl.acm.org/doi/10.1145/3132747.3132763", "title": "My VM is lighter (and safer) than your container", "date_modified": "2024-05-14T11:24:02.000Z" }, { "content_html": "Comments", "url": "https://marc.info/?l=openbsd-tech&m=171562561424289", "title": "Add sysctl to disable Nagle's algorithm (RFC 896 – Congestion Control)", "date_modified": "2024-05-14T07:51:04.000Z" }, { "content_html": "Comments", "url": "https://qmacro.org/blog/posts/2024/05/13/using-arg-in-a-dockerfile-beware-the-gotcha/", "title": "Using ARG in a Dockerfile – beware the gotcha", "date_modified": "2024-05-14T07:05:38.000Z" }, { "content_html": "Comments", "url": "https://www.cultured.systems/2024/04/24/Soft-delete/", "title": "Avoiding the soft delete anti-pattern", "date_modified": "2024-05-11T08:26:58.000Z" }, { "content_html": "Comments", "url": "https://www.flatcar.org/blog/2024/04/os-innovation-with-systemd-sysext/", "title": "Flatcar: OS Innovation with Systemd-Sysext", "date_modified": "2024-05-11T06:27:41.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40318542", "title": "Show HN: A web debugger an ex-Cloudflare team has been working on for 4 years", "date_modified": "2024-05-10T13:08:38.000Z" }, { "content_html": "Comments", "url": "https://buf.build/blog/protobuf-editions-are-here", "title": "Protobuf Editions are here: don't panic", "date_modified": "2024-05-09T20:58:45.000Z" }, { "content_html": "Comments", "url": "https://brooker.co.za/blog/2024/05/09/nagle.html", "title": "It's always TCP_NODELAY", "date_modified": "2024-05-09T17:54:49.000Z" }, { "content_html": "Comments", "url": "https://www.ellipsis.dev/", "title": "Show HN: Ellipsis – Automated PR reviews and bug fixes", "date_modified": "2024-05-09T16:14:47.000Z" }, { "content_html": "

Comments

", "url": "https://samuel.dionne-riel.com/blog/2024/05/07/its-not-flakes-vs-channels.html", "title": "It's not about “Flakes vs. Channels”", "date_modified": "2024-05-08T23:20:34.000Z" }, { "content_html": "

Comments

", "url": "https://antonz.org/sqlite-generated-columns/", "title": "Modern SQLite: Generated columns", "date_modified": "2024-05-08T13:53:24.000Z" }, { "content_html": "Comments", "url": "https://encore.dev/blog/encore-for-typescript", "title": "Encore: Distributed systems runtime for TypeScript, written in Rust", "date_modified": "2024-05-08T13:32:35.000Z" }, { "content_html": "

Comments

", "url": "https://www.oliviaking.com/inclusive-sans", "title": "Inclusive Sans, a text font designed for accessibility and readability", "date_modified": "2024-05-08T12:54:26.000Z" }, { "content_html": "Comments", "url": "https://fly.io/infra-log/", "title": "Fly.io Infra log: week-by-week record of what the team does", "date_modified": "2024-05-08T06:27:59.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40276768", "title": "Show HN: Peerdb Streams – Simple, native Postgres change data capture", "date_modified": "2024-05-06T17:00:42.000Z" }, { "content_html": "Comments", "url": "https://keygen.sh/blog/your-14-day-free-trial-aint-gonna-cut-it/", "title": "A fourteen-day free trial ain’t gonna cut it", "date_modified": "2024-05-06T13:50:10.000Z" }, { "content_html": "Comments", "url": "https://lix.systems/", "title": "Lix is a modern, delicious implementation of the Nix package manager", "date_modified": "2024-05-06T04:55:14.000Z" }, { "content_html": "Comments", "url": "https://j6b72.de/article/why-you-should-take-a-look-at-traefik/", "title": "Take a look at Traefik, even if you don't use containers", "date_modified": "2024-05-05T11:31:17.000Z" }, { "content_html": "Comments", "url": "https://depot.dev/blog/building-container-layers-from-scratch", "title": "Building Containers from Scratch: Layers", "date_modified": "2024-05-05T06:20:35.000Z" }, { "content_html": "Comments", "url": "https://chatgpt.com", "title": "OpenAI Bought Chatgpt.com", "date_modified": "2024-05-04T17:36:21.000Z" }, { "content_html": "

Comments

", "url": "https://deno.com/blog/building-deno-kv", "title": "Deno KV internals: building a database for the modern web", "date_modified": "2024-05-04T17:01:51.000Z" }, { "content_html": "Comments", "url": "https://old.reddit.com/r/wallstreetbets/comments/1cirikp/snowflake_isnt_worth_50_billion_dollars/", "title": "Snowflake's tech is not worth $50B dollars", "date_modified": "2024-05-03T22:30:31.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40231790", "title": "Ask HN: How do people create those sleek looking demos for startups?", "date_modified": "2024-05-02T01:31:30.000Z" }, { "content_html": "Comments", "url": "https://www.terminal.shop/", "title": "New startup sells coffee through SSH", "date_modified": "2024-05-01T18:26:33.000Z" }, { "content_html": "

The Kubernetes Container Runtime Interface (CRI)\nacts as the main connection between the kubelet\nand the Container Runtime.\nThose runtimes have to provide a gRPC server which has to\nfulfill a Kubernetes defined Protocol Buffer interface.\nThis API definition\nevolves over time, for example when contributors add new features or fields are\ngoing to become deprecated.

\n

In this blog post, I'd like to dive into the functionality and history of three\nextraordinary Remote Procedure Calls (RPCs), which are truly outstanding in\nterms of how they work: Exec, Attach and PortForward.

\n

Exec can be used to run dedicated commands within the container and stream\nthe output to a client like kubectl or\ncrictl. It also allows interaction with\nthat process using standard input (stdin), for example if users want to run a\nnew shell instance within an existing workload.

\n

Attach streams the output of the currently running process via standard I/O\nfrom the container to the client and also allows interaction with them. This is\nparticularly useful if users want to see what is going on in the container and\nbe able to interact with the process.

\n

PortForward can be utilized to forward a port from the host to the container\nto be able to interact with it using third party network tools. This allows it\nto bypass Kubernetes services\nfor a certain workload and interact with its network interface.

\n

What is so special about them?

\n

All RPCs of the CRI either use the gRPC unary calls\nfor communication or the server side streaming\nfeature (only GetContainerEvents right now). This means that mainly all RPCs\nretrieve a single client request and have to return a single server response.\nThe same applies to Exec, Attach, and PortForward, where their protocol definition\nlooks like this:

\n
// Exec prepares a streaming endpoint to execute a command in the container.\nrpc Exec(ExecRequest) returns (ExecResponse) {}\n
// Attach prepares a streaming endpoint to attach to a running container.\nrpc Attach(AttachRequest) returns (AttachResponse) {}\n
// PortForward prepares a streaming endpoint to forward ports from a PodSandbox.\nrpc PortForward(PortForwardRequest) returns (PortForwardResponse) {}\n

The requests carry everything required to allow the server to do the work,\nfor example, the ContainerId or command (Cmd) to be run in case of Exec.\nMore interestingly, all of their responses only contain a url:

\n
message ExecResponse {\n // Fully qualified URL of the exec streaming server.\n string url = 1;\n}\n
message AttachResponse {\n // Fully qualified URL of the attach streaming server.\n string url = 1;\n}\n
message PortForwardResponse {\n // Fully qualified URL of the port-forward streaming server.\n string url = 1;\n}\n

Why is it implemented like that? Well, the original design document\nfor those RPCs even predates Kubernetes Enhancements Proposals (KEPs)\nand was originally outlined back in 2016. The kubelet had a native\nimplementation for Exec, Attach, and PortForward before the\ninitiative to bring the functionality to the CRI started. Before that,\neverything was bound to Docker or the later abandoned\ncontainer runtime rkt.

\n

The CRI related design document also elaborates on the option to use native RPC\nstreaming for exec, attach, and port forward. The downsides outweighed this\napproach: the kubelet would still create a network bottleneck and future\nruntimes would not be free in choosing the server implementation details. Also,\nanother option that the Kubelet implements a portable, runtime-agnostic solution\nhas been abandoned over the final one, because this would mean another project\nto maintain which nevertheless would be runtime dependent.

\n

This means, that the basic flow for Exec, Attach and PortForward\nwas proposed to look like this:

\n
\n\"CRI\n
\n

Clients like crictl or the kubelet (via kubectl) request a new exec, attach or\nport forward session from the runtime using the gRPC interface. The runtime\nimplements a streaming server that also manages the active sessions. This\nstreaming server provides an HTTP endpoint for the client to connect to. The\nclient upgrades the connection to use the SPDY\nstreaming protocol or (in the future) to a WebSocket\nconnection and starts to stream the data back and forth.

\n

This implementation allows runtimes to have the flexibility to implement\nExec, Attach and PortForward the way they want, and also allows a\nsimple test path. Runtimes can change the underlying implementation to support\nany kind of feature without having a need to modify the CRI at all.

\n

Many smaller enhancements to this overall approach have been merged into\nKubernetes in the past years, but the general pattern has always stayed the\nsame. The kubelet source code transformed into a reusable library,\nwhich is nowadays usable from container runtimes to implement the basic\nstreaming capability.

\n

How does the streaming actually work?

\n

At a first glance, it looks like all three RPCs work the same way, but that's\nnot the case. It's possible to group the functionality of Exec and\nAttach, while PortForward follows a distinct internal protocol\ndefinition.

\n

Exec and Attach

\n

Kubernetes defines Exec and Attach as remote commands, where its\nprotocol definition exists in five different versions:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
#VersionNote
1channel.k8s.ioInitial (unversioned) SPDY sub protocol (#13394, #13395)
2v2.channel.k8s.ioResolves the issues present in the first version (#15961)
3v3.channel.k8s.ioAdds support for resizing container terminals (#25273)
4v4.channel.k8s.ioAdds support for exit codes using JSON errors (#26541)
5v5.channel.k8s.ioAdds support for a CLOSE signal (#119157)
\n

On top of that, there is an overall effort to replace the SPDY transport\nprotocol using WebSockets as part KEP #4006.\nRuntimes have to satisfy those protocols over their life cycle to stay up to\ndate with the Kubernetes implementation.

\n

Let's assume that a client uses the latest (v5) version of the protocol as\nwell as communicating over WebSockets. In that case, the general flow would be:

\n
    \n
  1. \n

    The client requests an URL endpoint for Exec or Attach using the CRI.

    \n\n
    2. \n
  2. \n

    The client connects to that URL, upgrades the connection to establish\na WebSocket, and starts to stream data.

    \n\n

    If stdin is required, then the server needs to listen for that as well and\nredirect it to the corresponding process.

    \n
    3. \n
\n

Interpreting data for the defined protocol is fairly simple: The first\nbyte of every input and output packet defines\nthe actual stream:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
First ByteTypeDescription
0standard inputData streamed from stdin
1standard outputData streamed to stdout
2standard errorData streamed to stderr
3stream errorA streaming error occurred
4stream resizeA terminal resize event
255stream closeStream should be closed (for WebSockets)
\n

How should runtimes now implement the streaming server methods for Exec and\nAttach by using the provided kubelet library? The key is that the streaming\nserver implementation in the kubelet outlines an interface\ncalled Runtime which has to be fulfilled by the actual container runtime if it\nwants to use that library:

\n
// Runtime is the interface to execute the commands and provide the streams.\ntype Runtime interface {\n Exec(ctx context.Context, containerID string, cmd []string, in io.Reader, out, err io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize) error\n Attach(ctx context.Context, containerID string, in io.Reader, out, err io.WriteCloser, tty bool, resize <-chan remotecommand.TerminalSize) error\n PortForward(ctx context.Context, podSandboxID string, port int32, stream io.ReadWriteCloser) error\n}\n

Everything related to the protocol interpretation is\nalready in place and runtimes only have to implement the actual Exec and\nAttach logic. For example, the container runtime CRI-O\ndoes it like this pseudo code:

\n
func (s StreamService) Exec(\n ctx context.Context,\n containerID string,\n cmd []string,\n stdin io.Reader, stdout, stderr io.WriteCloser,\n tty bool,\n resizeChan <-chan remotecommand.TerminalSize,\n) error {\n // Retrieve the container by the provided containerID\n // …\n\n // Update the container status and verify that the workload is running\n // …\n\n // Execute the command and stream the data\n return s.runtimeServer.Runtime().ExecContainer(\n s.ctx, c, cmd, stdin, stdout, stderr, tty, resizeChan,\n )\n}\n

PortForward

\n

Forwarding ports to a container works a bit differently when comparing it to\nstreaming IO data from a workload. The server still has to provide a URL\nendpoint for the client to connect to, but then the container runtime has to\nenter the network namespace of the container, allocate the port as well as\nstream the data back and forth. There is no simple protocol definition available\nlike for Exec or Attach. This means that the client will stream the\nplain SPDY frames (with or without an additional WebSocket connection) which can\nbe interpreted using libraries like moby/spdystream.

\n

Luckily, the kubelet library already provides the PortForward interface method\nwhich has to be implemented by the runtime. CRI-O does that by (simplified):

\n
func (s StreamService) PortForward(\n ctx context.Context,\n podSandboxID string,\n port int32,\n stream io.ReadWriteCloser,\n) error {\n // Retrieve the pod sandbox by the provided podSandboxID\n sandboxID, err := s.runtimeServer.PodIDIndex().Get(podSandboxID)\n sb := s.runtimeServer.GetSandbox(sandboxID)\n // …\n\n // Get the network namespace path on disk for that sandbox\n netNsPath := sb.NetNsPath()\n // …\n\n // Enter the network namespace and stream the data\n return s.runtimeServer.Runtime().PortForwardContainer(\n ctx, sb.InfraContainer(), netNsPath, port, stream,\n )\n}\n

Future work

\n

The flexibility Kubernetes provides for the RPCs Exec, Attach and\nPortForward is truly outstanding compared to other methods. Nevertheless,\ncontainer runtimes have to keep up with the latest and greatest implementations\nto support those features in a meaningful way. The general effort to support\nWebSockets is not only a plain Kubernetes thing, it also has to be supported by\ncontainer runtimes as well as clients like crictl.

\n

For example, crictl v1.30 features a new --transport flag for the\nsubcommands exec, attach and port-forward\n(#1383,\n#1385)\nto allow choosing between websocket and spdy.

\n

CRI-O is going an experimental path by moving the streaming server\nimplementation into conmon-rs\n(a substitute for the container monitor conmon). conmon-rs is\na Rust implementation of the original container\nmonitor and allows streaming WebSockets directly using supported libraries\n(#2070). The major benefit\nof this approach is that CRI-O does not even have to be running while conmon-rs\ncan keep active Exec, Attach and PortForward sessions open. The\nsimplified flow when using crictl directly will then look like this:

\n
\n
\nsequenceDiagram\nautonumber\nparticipant crictl\nparticipant runtime as Container Runtime\nparticipant conmon-rs\nNote over crictl,runtime: Container Runtime Interface (CRI)\ncrictl->>runtime: Exec, Attach, PortForward\nNote over runtime,conmon-rs: Cap’n Proto\nruntime->>conmon-rs: Serve Exec, Attach, PortForward\nconmon-rs->>runtime: HTTP endpoint (URL)\nruntime->>crictl: Response URL\ncrictl-->>conmon-rs: Connection upgrade to WebSocket\nconmon-rs-)crictl: Stream data\n
\n
\n\n

All of those enhancements require iterative design decisions, while the original\nwell-conceived implementation acts as the foundation for those. I really hope\nyou've enjoyed this compact journey through the history of CRI RPCs. Feel free\nto reach out to me anytime for suggestions or feedback using the\nofficial Kubernetes Slack.

", "url": "https://kubernetes.io/blog/2024/05/01/cri-streaming-explained/", "title": "Container Runtime Interface streaming explained", "date_modified": "2024-05-01T00:00:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/borgo-lang/borgo", "title": "Borgo is a statically typed language that compiles to Go", "date_modified": "2024-04-30T15:13:22.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=ddTV12hErTc", "title": "Rabbit R1: Barely Reviewable [video]", "date_modified": "2024-04-30T00:44:13.000Z" }, { "content_html": "Comments", "url": "https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1", "title": "How an empty S3 bucket can make your AWS bill explode", "date_modified": "2024-04-29T19:42:23.000Z" }, { "content_html": "

Comments

", "url": "https://aux.computer/", "title": "aux.computer: an alternative to the Nix ecosystem", "date_modified": "2024-04-29T19:20:59.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2024-04-29-github-copilot-workspace/", "title": "GitHub Copilot Workspace: Copilot-native developer environment", "date_modified": "2024-04-29T16:03:03.000Z" }, { "content_html": "Comments", "url": "https://www.warpstream.com/blog/tiered-storage-wont-fix-kafka", "title": "Tiered storage won't fix Kafka", "date_modified": "2024-04-29T14:54:38.000Z" }, { "content_html": "Comments", "url": "https://github.com/remorses/docker-phobia", "title": "Show HN: Docker-phobia: Analyze Docker image size with a treemap", "date_modified": "2024-04-28T13:01:12.000Z" }, { "content_html": "Comments", "url": "https://postgres.ai/blog/20220525-common-db-schema-change-mistakes", "title": "Common DB schema change mistakes in Postgres", "date_modified": "2024-04-28T07:30:40.000Z" }, { "content_html": "Comments", "url": "https://www.devicu.com/blog/software-supply-chan-security", "title": "Software Supply Chain Security", "date_modified": "2024-04-27T03:31:15.000Z" }, { "content_html": "Comments", "url": "https://railsdesigner.com/hidden-gems-tailwind/", "title": "Gems of Tailwind CSS", "date_modified": "2024-04-26T16:36:21.000Z" }, { "content_html": "Comments", "url": "https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/optimizing-your-programs-for-arm-platforms", "title": "Optimizing your programs for Arm platforms", "date_modified": "2024-04-26T12:10:59.000Z" }, { "content_html": "Comments", "url": "https://www.osohq.com/post/logic-language-distributed-sql-queries", "title": "A Logic Language for Distributed SQL Queries", "date_modified": "2024-04-25T21:13:52.000Z" }, { "content_html": "Comments", "url": "https://www.stainlessapi.com/blog/announcing-the-stainless-sdk-generator", "title": "The Stainless SDK Generator", "date_modified": "2024-04-24T16:34:35.000Z" }, { "content_html": "

Comments

", "url": "https://it-notes.dragas.net/2024/04/22/the-doubled-edge-sword-of-docker/", "title": "Double Edged Sword of Docker: Balancing Benefits and Risks", "date_modified": "2024-04-23T15:19:32.000Z" }, { "content_html": "Comments", "url": "https://blacksmith.sh/blog/going-through-yc-winter-24", "title": "What it was like going through YC W24", "date_modified": "2024-04-22T18:55:59.000Z" }, { "content_html": "Comments", "url": "https://community.fly.io/t/some-volumes-were-slow-and-we-figured-out-why/19394", "title": "Some Volumes Were Slow and We Figured Out Why", "date_modified": "2024-04-19T22:47:51.000Z" }, { "content_html": "Comments", "url": "https://github.com/1buran/rHttp", "title": "RHttp: REPL for HTTP", "date_modified": "2024-04-17T21:17:58.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=40062263", "title": "Ask HN: What software sparks joy when using?", "date_modified": "2024-04-17T09:26:47.000Z" }, { "content_html": "Comments", "url": "https://csvbase.com/blog/8", "title": "Caching secrets of the HTTP elders, part 1", "date_modified": "2024-04-17T05:19:55.000Z" }, { "content_html": "

Comments

", "url": "https://effect.website/blog/effect-3.0", "title": "Effect 3.0", "date_modified": "2024-04-16T15:44:15.000Z" }, { "content_html": "Comments", "url": "https://www.osohq.com/post/distributed-authorization", "title": "Launching Distributed Authorization", "date_modified": "2024-04-16T14:51:48.000Z" }, { "content_html": "Comments", "url": "https://www.osohq.com/post/distributed-authorization", "title": "Distributed Authorization", "date_modified": "2024-04-16T14:51:48.000Z" }, { "content_html": "Comments", "url": "https://pmf.firstround.com/levels", "title": "Product-Market Fit Isn't a Black Box – A New Framework to Help B2B Founders", "date_modified": "2024-04-16T13:48:55.000Z" }, { "content_html": "Comments", "url": "https://ssg.dev/ipv6-for-the-remotely-interested-af214dd06aa7?gi=79f1ddf22e27", "title": "IPv6 for the Remotely Interested", "date_modified": "2024-04-16T08:39:04.000Z" }, { "content_html": "Comments", "url": "https://www.ralfj.de/blog/2024/04/14/bubblebox.html", "title": "Sandboxing All the Things with Flatpak and BubbleBox", "date_modified": "2024-04-14T19:04:19.000Z" }, { "content_html": "Comments", "url": "https://www.theguardian.com/world/live/2024/apr/13/iran-launches-drone-attack-against-israel", "title": "Iran launches drone attack against Israel", "date_modified": "2024-04-13T20:59:45.000Z" }, { "content_html": "Comments", "url": "https://www.sequoiacap.com/article/pmf-framework/", "title": "The Arc Product-Market Fit Framework", "date_modified": "2024-04-13T04:55:41.000Z" }, { "content_html": "Comments", "url": "https://kenkantzer.com/lessons-after-a-half-billion-gpt-tokens/", "title": "Lessons after a Half-billion GPT Tokens", "date_modified": "2024-04-12T17:06:38.000Z" }, { "content_html": "Comments", "url": "https://blog.allegro.tech/2024/04/ten-years-microservices.html", "title": "Ten Years and Counting: My Affair with Microservices", "date_modified": "2024-04-12T08:52:01.000Z" }, { "content_html": "Comments", "url": "https://github.com/mhx/dwarfs", "title": "DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System", "date_modified": "2024-04-12T02:04:54.000Z" }, { "content_html": "Comments", "url": "https://stiankri.substack.com/p/the-open-secret-about-confidential", "title": "The Open Secret about Confidential Computing", "date_modified": "2024-04-09T16:30:37.000Z" }, { "content_html": "Comments", "url": "https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu/", "title": "Google Axion Processors – Arm-based CPUs designed for the data center", "date_modified": "2024-04-09T12:12:51.000Z" }, { "content_html": "Comments", "url": "https://tembo.io/blog/managed-postgres-rust", "title": "Building a Managed Postgres Service in Rust", "date_modified": "2024-04-08T07:40:40.000Z" }, { "content_html": "Comments", "url": "https://neon.tech/blog/architecture-decisions-in-neon", "title": "Architecture decisions in Neon, a serverless Postgres service", "date_modified": "2024-04-08T00:04:44.000Z" }, { "content_html": "Comments", "url": "https://github.com/rpgp/rpgp", "title": "Rpgp: Pure Rust implementation of OpenPGP", "date_modified": "2024-04-07T16:50:30.000Z" }, { "content_html": "Comments", "url": "https://github.com/stackframe-projects/pgmock", "title": "Show HN: I open-sourced the in-memory PostgreSQL I built at work for E2E tests", "date_modified": "2024-04-07T13:13:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/tazjin/nix-1p/blob/master/README.md", "title": "Nix – A One Pager", "date_modified": "2024-04-07T02:44:42.000Z" }, { "content_html": "Comments", "url": "https://breadchris.com/blog/my-favorite-button-on-the-internet/", "title": "My favorite button on the internet", "date_modified": "2024-04-06T17:19:14.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/OpenTofuOrg/status/1776398008558493991", "title": "HashiCorp: OpenTofu was not respecting the terms of its BSL license", "date_modified": "2024-04-06T15:06:41.000Z" }, { "content_html": "Comments", "url": "https://lukeshaughnessy.medium.com/infrastructure-as-code-is-not-the-answer-cfaf4882dcba", "title": "Infrastructure as Code Is Not the Answer – By Luke Shaughnessy", "date_modified": "2024-04-05T10:16:38.000Z" }, { "content_html": "Comments", "url": "https://www.datanami.com/2024/04/01/opentelemetry-is-too-complicated-victoriametrics-says/", "title": "OpenTelemetry Is Too Complicated, VictoriaMetrics Says", "date_modified": "2024-04-05T08:40:41.000Z" }, { "content_html": "Comments", "url": "https://www.bytebase.com/blog/features-i-wish-postgres-had/", "title": "Features I wish PostgreSQL had as a developer", "date_modified": "2024-04-05T07:06:21.000Z" }, { "content_html": "Comments", "url": "https://leontrolski.github.io/pglockpy.html", "title": "Postgres locks explorer", "date_modified": "2024-04-05T05:45:36.000Z" }, { "content_html": "Comments", "url": "https://vercel.com/blog/improved-infrastructure-pricing", "title": "Vercel: Improved Infrastructure Pricing", "date_modified": "2024-04-04T16:52:01.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/kelseyhightower/status/1775913633064894669", "title": "Developers, what marketing strategies work on you?", "date_modified": "2024-04-04T16:26:28.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=39930908", "title": "Show HN: Managed GitHub Actions Runners for AWS", "date_modified": "2024-04-04T14:32:20.000Z" }, { "content_html": "

Comments

", "url": "https://clickhouse.com/blog/building-a-logging-platform-with-clickhouse-and-saving-millions-over-datadog", "title": "How we Built a 19 PiB Logging Platform with ClickHouse and Saved Millions", "date_modified": "2024-04-03T14:41:34.000Z" }, { "content_html": "Comments", "url": "https://github.com/0verread/goralim", "title": "Show HN: Goralim - a rate limiting pkg for Go to handle distributed workloads", "date_modified": "2024-04-03T05:52:35.000Z" }, { "content_html": "

Comments

", "url": "https://blog.jetbrains.com/go/2024/03/26/data-flow-analysis-for-go/", "title": "Data Flow Analysis for Go", "date_modified": "2024-04-02T18:29:40.000Z" }, { "content_html": "

Comments

", "url": "https://linderud.dev/blog/nixos-is-not-reproducible/", "title": "NixOS is not reproducible", "date_modified": "2024-04-02T17:24:55.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/python-workers", "title": "Python Cloudflare Workers", "date_modified": "2024-04-02T13:20:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/unikraft/unikraft", "title": "KraftCloud", "date_modified": "2024-04-02T06:58:36.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=39902949", "title": "Unikraft Launches KraftCloud: Never Pay for Idle Again", "date_modified": "2024-04-02T06:35:03.000Z" }, { "content_html": "Comments", "url": "https://docs.kernel.org/userspace-api/landlock.html", "title": "Landlock: Unprivileged Access Control", "date_modified": "2024-03-30T16:06:57.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/tls-outage-20240307", "title": "About the Tailscale.com outage on March 7, 2024", "date_modified": "2024-03-30T15:34:58.000Z" }, { "content_html": "Comments", "url": "https://externals.io/message/122811", "title": "PHP: In light of xz backdoor, consider removing autogenerated files from release", "date_modified": "2024-03-30T11:58:23.000Z" }, { "content_html": "Comments", "url": "https://tableplus.com/blog/2024/03/how-we-deal-with-ddos.html", "title": "We are under DDoS attack and we do nothing", "date_modified": "2024-03-30T07:35:30.000Z" }, { "content_html": "Comments", "url": "https://bpfman.io/main/", "title": "Bpfman: An eBPF Manager", "date_modified": "2024-03-30T04:49:45.000Z" }, { "content_html": "Comments", "url": "https://blog.ekern.me/2024/03/28/do-you-really-need-kubernetes.html", "title": "Do you really need Kubernetes?", "date_modified": "2024-03-29T20:22:04.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/966869/4077c682c91ab5ab/", "title": "Radicle: Peer-to-Peer Collaboration with Git", "date_modified": "2024-03-29T20:12:57.000Z" }, { "content_html": "Comments", "url": "https://github.com/pojntfx/weron", "title": "Overlay Networks Based on WebRTC", "date_modified": "2024-03-29T17:00:07.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=39866083", "title": "Beware Scaling on AWS in Early Days", "date_modified": "2024-03-29T16:41:12.000Z" }, { "content_html": "Comments", "url": "https://radicle.xyz/guides/protocol", "title": "How Radicle Works Under the Hood", "date_modified": "2024-03-27T09:47:32.000Z" }, { "content_html": "Comments", "url": "https://www.annwan.me/computers/what-why-how-containers/", "title": "The What, Why and How of Containers", "date_modified": "2024-03-27T09:00:50.000Z" }, { "content_html": "Comments", "url": "https://blog.podman.io/2024/03/podman-5-0-has-been-released/", "title": "Podman 5.0 has been released", "date_modified": "2024-03-26T18:20:55.000Z" }, { "content_html": "Comments", "url": "https://saltycrackers.dev/posts/bye-bye-async-false/", "title": "`async: false` is the worst", "date_modified": "2024-03-26T17:52:45.000Z" }, { "content_html": "Comments", "url": "https://goto.buzzsprout.com/1714721/14674925", "title": "Elixir's Impact: Shaping the Evolution of Erlang", "date_modified": "2024-03-26T09:14:13.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/965631/f3db6d3dde3f83cb/", "title": "Nix at Scale", "date_modified": "2024-03-26T02:52:30.000Z" }, { "content_html": "Comments", "url": "https://nitric.io/blog/terraform-comparison-demo", "title": "Speeding up Azure development by not using Terraform", "date_modified": "2024-03-25T19:35:44.000Z" }, { "content_html": "Comments", "url": "https://github.com/pg-sharding/spqr/discussions/569", "title": "SPQR 1.3.0: a production-ready system for horizontal scaling of PostgreSQL", "date_modified": "2024-03-25T11:24:39.000Z" }, { "content_html": "Comments", "url": "https://divy.work/turbocall.html", "title": "Turbocall – Just-in-time compiler for Deno FFI", "date_modified": "2024-03-25T07:41:48.000Z" }, { "content_html": "Comments", "url": "https://yoyo-code.com/build-time-is-collective-responsibility/", "title": "Build time is a collective responsibility", "date_modified": "2024-03-24T11:53:22.000Z" }, { "content_html": "Comments", "url": "https://play.d2lang.com/", "title": "D2 Playground", "date_modified": "2024-03-24T06:56:58.000Z" }, { "content_html": "Comments", "url": "https://oxide.computer/", "title": "Oxide Cloud Computer. No Cables. No Assembly. Just Cloud", "date_modified": "2024-03-24T00:20:40.000Z" }, { "content_html": "Comments", "url": "https://ntex.rs/", "title": "Ntex: Powerful, pragmatic, fast framework for composable networking services", "date_modified": "2024-03-23T17:12:46.000Z" }, { "content_html": "

Comments

", "url": "https://m.youtube.com/watch?v=6iviTZfiLGU", "title": "Nix The Planet", "date_modified": "2024-03-22T20:14:47.000Z" }, { "content_html": "Comments", "url": "https://pack.ac/note/pack", "title": "Pack: A New Container Format for Compressed Files", "date_modified": "2024-03-22T19:11:45.000Z" }, { "content_html": "Comments", "url": "https://blog.allegro.tech/2024/03/kafka-performance-analysis.html", "title": "Tackling Tail Latency with eBPF", "date_modified": "2024-03-22T09:43:39.000Z" }, { "content_html": "Comments", "url": "https://github.com/redis/redis/blob/unstable/LICENSE.txt", "title": "Redis License Changed", "date_modified": "2024-03-21T06:12:04.000Z" }, { "content_html": "Comments", "url": "https://benw.is/posts/how-i-improved-my-rust-compile-times-by-seventy-five-percent", "title": "I Improved My Rust Compile Times", "date_modified": "2024-03-20T04:14:35.000Z" }, { "content_html": "

Comments

", "url": "https://yzena.com/2024/03/build-system-schism-the-curse-of-meta-build-systems/", "title": "Build System Schism: The Curse Of Meta Build Systems", "date_modified": "2024-03-20T01:12:07.000Z" }, { "content_html": "Comments", "url": "https://www.stardewvalley.net/stardew-valley-1-6-update-full-changelog/", "title": "Stardew Valley 1.6 Changelog", "date_modified": "2024-03-19T19:21:55.000Z" }, { "content_html": "Comments", "url": "https://github.com/failsafe-go/failsafe-go", "title": "Fault tolerance and resilience patterns for Go", "date_modified": "2024-03-19T18:19:32.000Z" }, { "content_html": "Comments", "url": "https://causal.app", "title": "Show HN: Causal 2.0 – Modern Financial Planning for Startups", "date_modified": "2024-03-19T14:06:25.000Z" }, { "content_html": "Comments", "url": "https://rysana.com/inversion", "title": "Inversion: Fast, Reliable Structured LLMs", "date_modified": "2024-03-18T21:10:38.000Z" }, { "content_html": "Comments", "url": "https://xata.io/blog/serverless-postgres-platform", "title": "Xata, a new serverless Postgres platform", "date_modified": "2024-03-18T18:24:01.000Z" }, { "content_html": "Comments", "url": "https://www.nngroup.com/articles/homepage-design-principles/", "title": "Homepage Design: 5 Fundamental Principles", "date_modified": "2024-03-17T19:25:33.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=39729307", "title": "Ask HN: Books that gave you different perspective on religion", "date_modified": "2024-03-16T20:39:31.000Z" }, { "content_html": "Comments", "url": "https://blog.hathora.dev/our-bare-metal-journey/", "title": "Hathora's Bare Metal Journey", "date_modified": "2024-03-15T21:17:06.000Z" }, { "content_html": "Comments", "url": "https://xeiaso.net/talks/2024/nix-docker-build/", "title": "Nix is a better Docker image builder than Docker's image builder", "date_modified": "2024-03-15T19:56:47.000Z" }, { "content_html": "Comments", "url": "https://matduggan.com/iam-is-the-worst/", "title": "IAM Is the Worst", "date_modified": "2024-03-15T10:55:22.000Z" }, { "content_html": "Comments", "url": "https://gopacker.dev/docs/", "title": "Programs written in Golang have no secrets", "date_modified": "2024-03-15T07:07:46.000Z" }, { "content_html": "Comments", "url": "https://go.dev/blog/execution-traces-2024", "title": "More powerful Go execution traces", "date_modified": "2024-03-14T20:30:17.000Z" }, { "content_html": "Comments", "url": "https://nanos.org", "title": "Nanos – A Unikernel", "date_modified": "2024-03-13T23:43:32.000Z" }, { "content_html": "

Comments

", "url": "https://medium.com/innovationendeavors/s3-as-the-universal-infrastructure-backend-a104a8cc6991", "title": "S3 as the universal infrastructure backend", "date_modified": "2024-03-13T19:46:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/flox/flox", "title": "Show HN: Flox 1.0 – Open-source dev env as code with Nix", "date_modified": "2024-03-13T15:44:29.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/jit-wireguard-peers/", "title": "JIT WireGuard", "date_modified": "2024-03-13T06:13:09.000Z" }, { "content_html": "Comments", "url": "https://calpaterson.com/s3.html", "title": "S3 is files, but not a filesystem", "date_modified": "2024-03-10T04:11:26.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/seshubon/status/1765870717844050221", "title": "Claude Sonnet and Pi AI give exact same response to a prompt", "date_modified": "2024-03-08T13:03:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/NilsIrl/dockerc", "title": "Show HN: dockerc – Docker image to static executable \"compiler\"", "date_modified": "2024-03-06T19:55:22.000Z" }, { "content_html": "Comments", "url": "https://yolken.net/blog/end-of-airplanedev", "title": "The End of Airplane.dev", "date_modified": "2024-03-06T18:16:58.000Z" }, { "content_html": "

The Linux kernel is the heart of many modern production systems. It decides when any code is allowed to run and which programs/users can access which resources. It manages memory, mediates access to hardware, and does a bulk of work under the hood on behalf of programs running on top. Since the kernel is always involved in any code execution, it is in the best position to protect the system from malicious programs, enforce the desired system security policy, and provide security features for safer production environments.

\n

In this post, we will review some Linux kernel security configurations we use at Cloudflare and how they help to block or minimize a potential system compromise.

\n

Secure boot

\n

When a machine (either a laptop or a server) boots, it goes through several boot stages:

\n

\n

Within a secure boot architecture each stage from the above diagram verifies the integrity of the next stage before passing execution to it, thus forming a so-called secure boot chain. This way “trustworthiness” is extended to every component in the boot chain, because if we verified the code integrity of a particular stage, we can trust this code to verify the integrity of the next stage.

\n

We have previously covered how Cloudflare implements secure boot in the initial stages of the boot process. In this post, we will focus on the Linux kernel.

\n

Secure boot is the cornerstone of any operating system security mechanism. The Linux kernel is the primary enforcer of the operating system security configuration and policy, so we have to be sure that the Linux kernel itself has not been tampered with. In our previous post about secure boot we showed how we use UEFI Secure Boot to ensure the integrity of the Linux kernel.

\n

But what happens next? After the kernel gets executed, it may try to load additional drivers, or as they are called in the Linux world, kernel modules. And kernel module loading is not confined just to the boot process. A module can be loaded at any time during runtime — a new device being plugged in and a driver is needed, some additional extensions in the networking stack are required (for example, for fine-grained firewall rules), or just manually by the system administrator.

\n

However, uncontrolled kernel module loading might pose a significant risk to system integrity. Unlike regular programs, which get executed as user space processes, kernel modules are pieces of code which get injected and executed directly in the Linux kernel address space. There is no separation between the code and data in different kernel modules and core kernel subsystems, so everything can access everything. This means that a rogue kernel module can completely nullify the trustworthiness of the operating system and make secure boot useless. As an example, consider a simple Debian 12 (Bookworm installation), but with SELinux configured and enforced:

\n
ignat@dev:~$ lsb_release --all\nNo LSB modules are available.\nDistributor ID:\tDebian\nDescription:\tDebian GNU/Linux 12 (bookworm)\nRelease:\t12\nCodename:\tbookworm\nignat@dev:~$ uname -a\nLinux dev 6.1.0-18-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux\nignat@dev:~$ sudo getenforce\nEnforcing\n
\n

Now we need to do some research. First, we see that we’re running 6.1.76 Linux Kernel. If we explore the source code, we would see that inside the kernel, the SELinux configuration is stored in a singleton structure, which is defined as follows:

\n
struct selinux_state {\n#ifdef CONFIG_SECURITY_SELINUX_DISABLE\n\tbool disabled;\n#endif\n#ifdef CONFIG_SECURITY_SELINUX_DEVELOP\n\tbool enforcing;\n#endif\n\tbool checkreqprot;\n\tbool initialized;\n\tbool policycap[__POLICYDB_CAP_MAX];\n\n\tstruct page *status_page;\n\tstruct mutex status_lock;\n\n\tstruct selinux_avc *avc;\n\tstruct selinux_policy __rcu *policy;\n\tstruct mutex policy_mutex;\n} __randomize_layout;\n
\n

From the above, we can see that if the kernel configuration has CONFIG_SECURITY_SELINUX_DEVELOP enabled, the structure would have a boolean variable enforcing, which controls the enforcement status of SELinux at runtime. This is exactly what the above $ sudo getenforce command returns. We can double check that the Debian kernel indeed has the configuration option enabled:

\n
ignat@dev:~$ grep CONFIG_SECURITY_SELINUX_DEVELOP /boot/config-`uname -r`\nCONFIG_SECURITY_SELINUX_DEVELOP=y\n
\n

Good! Now that we have a variable in the kernel, which is responsible for some security enforcement, we can try to attack it. One problem though is the __randomize_layout attribute: since CONFIG_SECURITY_SELINUX_DISABLE is actually not set for our Debian kernel, normally enforcing would be the first member of the struct. Thus if we know where the struct is, we immediately know the position of the enforcing flag. With __randomize_layout, during kernel compilation the compiler might place members at arbitrary positions within the struct, so it is harder to create generic exploits. But arbitrary struct randomization within the kernel may introduce performance impact, so is often disabled and it is disabled for the Debian kernel:

\n
ignat@dev:~$ grep RANDSTRUCT /boot/config-`uname -r`\nCONFIG_RANDSTRUCT_NONE=y\n
\n

We can also confirm the compiled position of the enforcing flag using the pahole tool and either kernel debug symbols, if available, or (on modern kernels, if enabled) in-kernel BTF information. We will use the latter:

\n
ignat@dev:~$ pahole -C selinux_state /sys/kernel/btf/vmlinux\nstruct selinux_state {\n\tbool                       enforcing;            /*     0     1 */\n\tbool                       checkreqprot;         /*     1     1 */\n\tbool                       initialized;          /*     2     1 */\n\tbool                       policycap[8];         /*     3     8 */\n\n\t/* XXX 5 bytes hole, try to pack */\n\n\tstruct page *              status_page;          /*    16     8 */\n\tstruct mutex               status_lock;          /*    24    32 */\n\tstruct selinux_avc *       avc;                  /*    56     8 */\n\t/* --- cacheline 1 boundary (64 bytes) --- */\n\tstruct selinux_policy *    policy;               /*    64     8 */\n\tstruct mutex               policy_mutex;         /*    72    32 */\n\n\t/* size: 104, cachelines: 2, members: 9 */\n\t/* sum members: 99, holes: 1, sum holes: 5 */\n\t/* last cacheline: 40 bytes */\n};\n
\n

So enforcing is indeed located at the start of the structure and we don’t even have to be a privileged user to confirm this.

\n

Great! All we need is the runtime address of the selinux_state variable inside the kernel:
\n(shell/bash)

\n
ignat@dev:~$ sudo grep selinux_state /proc/kallsyms\nffffffffbc3bcae0 B selinux_state\n
\n

With all the information, we can write an almost textbook simple kernel module to manipulate the SELinux state:

\n

Mymod.c:

\n
#include <linux/module.h>\n\nstatic int __init mod_init(void)\n{\n\tbool *selinux_enforce = (bool *)0xffffffffbc3bcae0;\n\t*selinux_enforce = false;\n\treturn 0;\n}\n\nstatic void mod_fini(void)\n{\n}\n\nmodule_init(mod_init);\nmodule_exit(mod_fini);\n\nMODULE_DESCRIPTION(\"A somewhat malicious module\");\nMODULE_AUTHOR(\"Ignat Korchagin <ignat@cloudflare.com>\");\nMODULE_LICENSE(\"GPL\");\n
\n

And the respective Kbuild file:

\n
obj-m := mymod.o\n
\n

With these two files we can build a full fledged kernel module according to the official kernel docs:

\n
ignat@dev:~$ cd mymod/\nignat@dev:~/mymod$ ls\nKbuild  mymod.c\nignat@dev:~/mymod$ make -C /lib/modules/`uname -r`/build M=$PWD\nmake: Entering directory '/usr/src/linux-headers-6.1.0-18-cloud-amd64'\n  CC [M]  /home/ignat/mymod/mymod.o\n  MODPOST /home/ignat/mymod/Module.symvers\n  CC [M]  /home/ignat/mymod/mymod.mod.o\n  LD [M]  /home/ignat/mymod/mymod.ko\n  BTF [M] /home/ignat/mymod/mymod.ko\nSkipping BTF generation for /home/ignat/mymod/mymod.ko due to unavailability of vmlinux\nmake: Leaving directory '/usr/src/linux-headers-6.1.0-18-cloud-amd64'\n
\n

If we try to load this module now, the system may not allow it due to the SELinux policy:

\n
ignat@dev:~/mymod$ sudo insmod mymod.ko\ninsmod: ERROR: could not load module mymod.ko: Permission denied\n
\n

We can workaround it by copying the module into the standard module path somewhere:

\n
ignat@dev:~/mymod$ sudo cp mymod.ko /lib/modules/`uname -r`/kernel/crypto/\n
\n

Now let’s try it out:

\n
ignat@dev:~/mymod$ sudo getenforce\nEnforcing\nignat@dev:~/mymod$ sudo insmod /lib/modules/`uname -r`/kernel/crypto/mymod.ko\nignat@dev:~/mymod$ sudo getenforce\nPermissive\n
\n

Not only did we disable the SELinux protection via a malicious kernel module, we did it quietly. Normal sudo setenforce 0, even if allowed, would go through the official selinuxfs interface and would emit an audit message. Our code manipulated the kernel memory directly, so no one was alerted. This illustrates why uncontrolled kernel module loading is very dangerous and that is why most security standards and commercial security monitoring products advocate for close monitoring of kernel module loading.

\n

But we don’t need to monitor kernel modules at Cloudflare. Let’s repeat the exercise on a Cloudflare production kernel (module recompilation skipped for brevity):

\n
ignat@dev:~/mymod$ uname -a\nLinux dev 6.6.17-cloudflare-2024.2.9 #1 SMP PREEMPT_DYNAMIC Mon Sep 27 00:00:00 UTC 2010 x86_64 GNU/Linux\nignat@dev:~/mymod$ sudo insmod /lib/modules/`uname -r`/kernel/crypto/mymod.ko\ninsmod: ERROR: could not insert module /lib/modules/6.6.17-cloudflare-2024.2.9/kernel/crypto/mymod.ko: Key was rejected by service\n
\n

We get a Key was rejected by service error when trying to load a module, and the kernel log will have the following message:

\n
ignat@dev:~/mymod$ sudo dmesg | tail -n 1\n[41515.037031] Loading of unsigned module is rejected\n
\n

This is because the Cloudflare kernel requires all the kernel modules to have a valid signature, so we don’t even have to worry about a malicious module being loaded at some point:

\n
ignat@dev:~$ grep MODULE_SIG_FORCE /boot/config-`uname -r`\nCONFIG_MODULE_SIG_FORCE=y\n
\n

For completeness it is worth noting that the Debian stock kernel also supports module signatures, but does not enforce it:

\n
ignat@dev:~$ grep MODULE_SIG /boot/config-6.1.0-18-cloud-amd64\nCONFIG_MODULE_SIG_FORMAT=y\nCONFIG_MODULE_SIG=y\n# CONFIG_MODULE_SIG_FORCE is not set\n…\n
\n

The above configuration means that the kernel will validate a module signature, if available. But if not - the module will be loaded anyway with a warning message emitted and the kernel will be tainted.

\n

Key management for kernel module signing

\n

Signed kernel modules are great, but it creates a key management problem: to sign a module we need a signing keypair that is trusted by the kernel. The public key of the keypair is usually directly embedded into the kernel binary, so the kernel can easily use it to verify module signatures. The private key of the pair needs to be protected and secure, because if it is leaked, anyone could compile and sign a potentially malicious kernel module which would be accepted by our kernel.

\n

But what is the best way to eliminate the risk of losing something? Not to have it in the first place! Luckily the kernel build system will generate a random keypair for module signing, if none is provided. At Cloudflare, we use that feature to sign all the kernel modules during the kernel compilation stage. When the compilation and signing is done though, instead of storing the key in a secure place, we just destroy the private key:

\n

\n

So with the above process:

\n
    \n
  1. The kernel build system generated a random keypair, compiles the kernel and modules
    2. \n
  2. The public key is embedded into the kernel image, the private key is used to sign all the modules
    3. \n
  3. The private key is destroyed
    4. \n
\n

With this scheme not only do we not have to worry about module signing key management, we also use a different key for each kernel we release to production. So even if a particular build process is hijacked and the signing key is not destroyed and potentially leaked, the key will no longer be valid when a kernel update is released.

\n

There are some flexibility downsides though, as we can’t “retrofit” a new kernel module for an already released kernel (for example, for a new piece of hardware we are adopting). However, it is not a practical limitation for us as we release kernels often (roughly every week) to keep up with a steady stream of bug fixes and vulnerability patches in the Linux Kernel.

\n

KEXEC

\n

KEXEC (or kexec_load()) is an interesting system call in Linux, which allows for one kernel to directly execute (or jump to) another kernel. The idea behind this is to switch/update/downgrade kernels faster without going through a full reboot cycle to minimize the potential system downtime. However, it was developed quite a while ago, when secure boot and system integrity was not quite a concern. Therefore its original design has security flaws and is known to be able to bypass secure boot and potentially compromise system integrity.

\n

We can see the problems just based on the definition of the system call itself:

\n
struct kexec_segment {\n\tconst void *buf;\n\tsize_t bufsz;\n\tconst void *mem;\n\tsize_t memsz;\n};\n...\nlong kexec_load(unsigned long entry, unsigned long nr_segments, struct kexec_segment *segments, unsigned long flags);\n
\n

So the kernel expects just a collection of buffers with code to execute. Back in those days there was not much desire to do a lot of data parsing inside the kernel, so the idea was to parse the to-be-executed kernel image in user space and provide the kernel with only the data it needs. Also, to switch kernels live, we need an intermediate program which would take over while the old kernel is shutting down and the new kernel has not yet been executed. In the kexec world this program is called purgatory. Thus the problem is evident: we give the kernel a bunch of code and it will happily execute it at the highest privilege level. But instead of the original kernel or purgatory code, we can easily provide code similar to the one demonstrated earlier in this post, which disables SELinux (or does something else to the kernel).

\n

At Cloudflare we have had kexec_load() disabled for some time now just because of this. The advantage of faster reboots with kexec comes with a (small) risk of improperly initialized hardware, so it was not worth using it even without the security concerns. However, kexec does provide one useful feature — it is the foundation of the Linux kernel crashdumping solution. In a nutshell, if a kernel crashes in production (due to a bug or some other error), a backup kernel (previously loaded with kexec) can take over, collect and save the memory dump for further investigation. This allows to more effectively investigate kernel and other issues in production, so it is a powerful tool to have.

\n

Luckily, since the original problems with kexec were outlined, Linux developed an alternative secure interface for kexec: instead of buffers with code it expects file descriptors with the to-be-executed kernel image and initrd and does parsing inside the kernel. Thus, only a valid kernel image can be supplied. On top of this, we can configure and require kexec to ensure the provided images are properly signed, so only authorized code can be executed in the kexec scenario. A secure configuration for kexec looks something like this:

\n
ignat@dev:~$ grep KEXEC /boot/config-`uname -r`\nCONFIG_KEXEC_CORE=y\nCONFIG_HAVE_IMA_KEXEC=y\n# CONFIG_KEXEC is not set\nCONFIG_KEXEC_FILE=y\nCONFIG_KEXEC_SIG=y\nCONFIG_KEXEC_SIG_FORCE=y\nCONFIG_KEXEC_BZIMAGE_VERIFY_SIG=y\n…\n
\n

Above we ensure that the legacy kexec_load() system call is disabled by disabling CONFIG_KEXEC, but still can configure Linux Kernel crashdumping via the new kexec_file_load() system call via CONFIG_KEXEC_FILE=y with enforced signature checks (CONFIG_KEXEC_SIG=y and CONFIG_KEXEC_SIG_FORCE=y).

\n

Note that stock Debian kernel has the legacy kexec_load() system call enabled and does not enforce signature checks for kexec_file_load() (similar to module signature checks):

\n
ignat@dev:~$ grep KEXEC /boot/config-6.1.0-18-cloud-amd64\nCONFIG_KEXEC=y\nCONFIG_KEXEC_FILE=y\nCONFIG_ARCH_HAS_KEXEC_PURGATORY=y\nCONFIG_KEXEC_SIG=y\n# CONFIG_KEXEC_SIG_FORCE is not set\nCONFIG_KEXEC_BZIMAGE_VERIFY_SIG=y\n…\n
\n

Kernel Address Space Layout Randomization (KASLR)

\n

Even on the stock Debian kernel if you try to repeat the exercise we described in the “Secure boot” section of this post after a system reboot, you will likely see it would fail to disable SELinux now. This is because we hardcoded the kernel address of the selinux_state structure in our malicious kernel module, but the address changed now:

\n
ignat@dev:~$ sudo grep selinux_state /proc/kallsyms\nffffffffb41bcae0 B selinux_state\n
\n

Kernel Address Space Layout Randomization (or KASLR) is a simple concept: it slightly and randomly shifts the kernel code and data on each boot:

\n

\n

This is to combat targeted exploitation (like the malicious module in this post) based on the knowledge of the location of internal kernel structures and code. It is especially useful for popular Linux distribution kernels, like the Debian one, because most users use the same binary and anyone can download the debug symbols and the System.map file with all the addresses of the kernel internals. Just to note: it will not prevent the module loading and doing harm, but it will likely not achieve the targeted effect of disabling SELinux. Instead, it will modify a random piece of kernel memory potentially causing the kernel to crash.

\n

Both the Cloudflare kernel and the Debian one have this feature enabled:

\n
ignat@dev:~$ grep RANDOMIZE_BASE /boot/config-`uname -r`\nCONFIG_RANDOMIZE_BASE=y\n
\n

Restricted kernel pointers

\n

While KASLR helps with targeted exploits, it is quite easy to bypass since everything is shifted by a single random offset as shown on the diagram above. Thus if the attacker knows at least one runtime kernel address, they can recover this offset by subtracting the runtime address from the compile time address of the same symbol (function or data structure) from the kernel’s System.map file. Once they know the offset, they can recover the addresses of all other symbols by adjusting them by this offset.

\n

Therefore, modern kernels take precautions not to leak kernel addresses at least to unprivileged users. One of the main tunables for this is the kptr_restrict sysctl. It is a good idea to set it at least to 1 to not allow regular users to see kernel pointers:
\n(shell/bash)

\n
ignat@dev:~$ sudo sysctl -w kernel.kptr_restrict=1\nkernel.kptr_restrict = 1\nignat@dev:~$ grep selinux_state /proc/kallsyms\n0000000000000000 B selinux_state\n
\n

Privileged users can still see the pointers:

\n
ignat@dev:~$ sudo grep selinux_state /proc/kallsyms\nffffffffb41bcae0 B selinux_state\n
\n

Similar to kptr_restrict sysctl there is also dmesg_restrict, which if set, would prevent regular users from reading the kernel log (which may also leak kernel pointers via its messages). While you need to explicitly set kptr_restrict sysctl to a non-zero value on each boot (or use some system sysctl configuration utility, like this one), you can configure dmesg_restrict initial value via the CONFIG_SECURITY_DMESG_RESTRICT kernel configuration option. Both the Cloudflare kernel and the Debian one enforce dmesg_restrict this way:

\n
ignat@dev:~$ grep CONFIG_SECURITY_DMESG_RESTRICT /boot/config-`uname -r`\nCONFIG_SECURITY_DMESG_RESTRICT=y\n
\n

Worth noting that /proc/kallsyms and the kernel log are not the only sources of potential kernel pointer leaks. There is a lot of legacy in the Linux kernel and [new sources are continuously being found and patched]. That’s why it is very important to stay up to date with the latest kernel bugfix releases.

\n

Lockdown LSM

\n

Linux Security Modules (LSM) is a hook-based framework for implementing security policies and Mandatory Access Control in the Linux Kernel. We have [covered our usage of another LSM module, BPF-LSM, previously].

\n

BPF-LSM is a useful foundational piece for our kernel security, but in this post we want to mention another useful LSM module we use — the Lockdown LSM. Lockdown can be in three states (controlled by the /sys/kernel/security/lockdown special file):

\n
ignat@dev:~$ cat /sys/kernel/security/lockdown\n[none] integrity confidentiality\n
\n

none is the state where nothing is enforced and the module is effectively disabled. When Lockdown is in the integrity state, the kernel tries to prevent any operation, which may compromise its integrity. We already covered some examples of these in this post: loading unsigned modules and executing unsigned code via KEXEC. But there are other potential ways (which are mentioned in the LSM’s man page), all of which this LSM tries to block. confidentiality is the most restrictive mode, where Lockdown will also try to prevent any information leakage from the kernel. In practice this may be too restrictive for server workloads as it blocks all runtime debugging capabilities, like perf or eBPF.

\n

Let’s see the Lockdown LSM in action. On a barebones Debian system the initial state is none meaning nothing is locked down:

\n
ignat@dev:~$ uname -a\nLinux dev 6.1.0-18-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux\nignat@dev:~$ cat /sys/kernel/security/lockdown\n[none] integrity confidentiality\n
\n

We can switch the system into the integrity mode:

\n
ignat@dev:~$ echo integrity | sudo tee /sys/kernel/security/lockdown\nintegrity\nignat@dev:~$ cat /sys/kernel/security/lockdown\nnone [integrity] confidentiality\n
\n

It is worth noting that we can only put the system into a more restrictive state, but not back. That is, once in integrity mode we can only switch to confidentiality mode, but not back to none:

\n
ignat@dev:~$ echo none | sudo tee /sys/kernel/security/lockdown\nnone\ntee: /sys/kernel/security/lockdown: Operation not permitted\n
\n

Now we can see that even on a stock Debian kernel, which as we discovered above, does not enforce module signatures by default, we cannot load a potentially malicious unsigned kernel module anymore:

\n
ignat@dev:~$ sudo insmod mymod/mymod.ko\ninsmod: ERROR: could not insert module mymod/mymod.ko: Operation not permitted\n
\n

And the kernel log will helpfully point out that this is due to Lockdown LSM:

\n
ignat@dev:~$ sudo dmesg | tail -n 1\n[21728.820129] Lockdown: insmod: unsigned module loading is restricted; see man kernel_lockdown.7\n
\n

As we can see, Lockdown LSM helps to tighten the security of a kernel, which otherwise may not have other enforcing bits enabled, like the stock Debian one.

\n

If you compile your own kernel, you can go one step further and set the initial state of the Lockdown LSM to be more restrictive than none from the start. This is exactly what we did for the Cloudflare production kernel:

\n
ignat@dev:~$ grep LOCK_DOWN /boot/config-6.6.17-cloudflare-2024.2.9\n# CONFIG_LOCK_DOWN_KERNEL_FORCE_NONE is not set\nCONFIG_LOCK_DOWN_KERNEL_FORCE_INTEGRITY=y\n# CONFIG_LOCK_DOWN_KERNEL_FORCE_CONFIDENTIALITY is not set\n
\n

Conclusion

\n

In this post we reviewed some useful Linux kernel security configuration options we use at Cloudflare. This is only a small subset, and there are many more available and even more are being constantly developed, reviewed, and improved by the Linux kernel community. We hope that this post will shed some light on these security features and that, if you haven’t already, you may consider enabling them in your Linux systems.

\n

Watch on Cloudflare TV

\n \n

Tune in for more news, announcements and thought-provoking discussions! Don't miss the full Security Week hub page.

", "url": "https://blog.cloudflare.com/linux-kernel-hardening", "title": "Linux kernel security tunables everyone should consider adopting", "date_modified": "2024-03-06T14:00:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/ortuman/nuke", "title": "Nuke: A memory arena implementation for Go", "date_modified": "2024-03-05T06:22:03.000Z" }, { "content_html": "Comments", "url": "https://github.com/harshadmanglani/polaris", "title": "Show HN: Workflow Orchestrator in Golang", "date_modified": "2024-03-04T17:21:42.000Z" }, { "content_html": "Comments", "url": "https://gleam.run/news/gleam-version-1/?q", "title": "Gleam Version 1", "date_modified": "2024-03-04T13:53:47.000Z" }, { "content_html": "Comments", "url": "https://paulbutler.org/2024/the-haters-guide-to-kubernetes/", "title": "The hater's guide to Kubernetes", "date_modified": "2024-03-03T16:44:05.000Z" }, { "content_html": "Comments", "url": "https://labs.scaleway.com/en/em-rv1/", "title": "Scaleway launches RISC-V servers", "date_modified": "2024-03-03T11:01:32.000Z" }, { "content_html": "Comments", "url": "https://www.jbernier.com/p?id=nodejs-stream-async-iterator", "title": "How to convert Node.js stream callback functions into an Async Iterator", "date_modified": "2024-03-03T03:06:52.000Z" }, { "content_html": "

Comments

", "url": "https://www.aspiring.dev/building-aws-sigv4-into-your-app/", "title": "How To Build AWS-Compatible APIs: AWS Sigv4", "date_modified": "2024-02-29T12:51:54.000Z" }, { "content_html": "Comments", "url": "https://pql.dev/", "title": "Pql, a pipelined query language that compiles to SQL (written in Go)", "date_modified": "2024-02-28T15:33:56.000Z" }, { "content_html": "Comments", "url": "https://github.com/cloudflare/pingora", "title": "Pingora: HTTP Server and Proxy Library, in Rust, by Cloudflare, Released", "date_modified": "2024-02-28T09:56:16.000Z" }, { "content_html": "Comments", "url": "https://github.com/cloudflare/pingora", "title": "Pingora: build fast, reliable and programmable networked systems", "date_modified": "2024-02-28T09:56:16.000Z" }, { "content_html": "Comments", "url": "https://testcontainers.com/", "title": "Testcontainers", "date_modified": "2024-02-27T23:15:57.000Z" }, { "content_html": "Comments", "url": "https://erikbern.com/2021/04/19/software-infrastructure-2.0-a-wishlist.html", "title": "Software Infrastructure 2.0: A Wishlist", "date_modified": "2024-02-27T11:23:17.000Z" }, { "content_html": "Comments", "url": "https://netflixtechblog.com/announcing-bpftop-streamlining-ebpf-performance-optimization-6a727c1ae2e5?gi=223d75ac1771", "title": "Bpftop: Streamlining eBPF performance optimization", "date_modified": "2024-02-27T01:04:09.000Z" }, { "content_html": "Comments", "url": "https://usefathom.com/blog/reduce-aws-bill", "title": "Reducing our AWS bill by $100k", "date_modified": "2024-02-27T00:16:47.000Z" }, { "content_html": "Comments", "url": "https://multi.app/", "title": "Multi – Multiplayer Collaboration for macOS", "date_modified": "2024-02-26T04:54:45.000Z" }, { "content_html": "Comments", "url": "https://www.baldurbjarnason.com/2024/disillusioned-with-deno/", "title": "Disillusioned with Deno", "date_modified": "2024-02-26T04:42:39.000Z" }, { "content_html": "Comments", "url": "https://www.aspiring.dev/fly-io-scheduler-part-2/", "title": "Building a Fly.io-Like Scheduler with Resource Requirements", "date_modified": "2024-02-25T12:53:00.000Z" }, { "content_html": "Comments", "url": "https://www.aspiring.dev/fly-io-scheduler-part-2/", "title": "Building a Fly.io-like scheduler with resource requirements", "date_modified": "2024-02-25T12:53:00.000Z" }, { "content_html": "Comments", "url": "https://resend.com/blog/incident-report-for-february-21-2024", "title": "Resend – Incident report for February 21st, 2024", "date_modified": "2024-02-23T03:00:48.000Z" }, { "content_html": "

Comments

", "url": "https://www.aspiring.dev/fly-io-scheduler-part-1/", "title": "Building a Fly.io-like Scheduler", "date_modified": "2024-02-22T12:19:24.000Z" }, { "content_html": "Comments", "url": "https://supernotes.app/open-source/sn-pro/", "title": "SN Pro Typeface", "date_modified": "2024-02-22T12:01:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/ory/kratos/releases/tag/v1.1.0", "title": "Auth0 OSS alternative Ory Kratos now with passwordless and SMS support", "date_modified": "2024-02-22T11:41:48.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/webhooks-suck", "title": "Webhooks suck, but here are alternatives", "date_modified": "2024-02-21T18:31:36.000Z" }, { "content_html": "Comments", "url": "https://grafana.com/blog/2024/02/21/how-the-open-source-caddy-server-uses-grafana-cloud-for-full-stack-observability/", "title": "How the open source Caddy server uses Grafana Cloud for full-stack observability", "date_modified": "2024-02-21T18:30:23.000Z" }, { "content_html": "Comments", "url": "https://remix.run/blog/remix-vite-stable", "title": "Remix Vite Is Now Stable", "date_modified": "2024-02-21T13:52:01.000Z" }, { "content_html": "Comments", "url": "https://github.com/readysettech/readyset", "title": "Readyset: A MySQL and Postgres wire-compatible caching layer", "date_modified": "2024-02-21T10:09:57.000Z" }, { "content_html": "Comments", "url": "http://databasearchitects.blogspot.com/2024/02/ssds-have-become-ridiculously-fast.html", "title": "SSDs have become fast, except in the cloud", "date_modified": "2024-02-20T16:59:43.000Z" }, { "content_html": "Comments", "url": "http://arighi.blogspot.com/2024/02/writing-scheduler-for-linux-in-rust.html", "title": "Writing a scheduler for Linux in Rust that runs in user-space", "date_modified": "2024-02-20T15:26:11.000Z" }, { "content_html": "Comments", "url": "https://liftingstones.org/articles/stonelifting-etiquette", "title": "Stonelifting Etiquette", "date_modified": "2024-02-20T11:19:35.000Z" }, { "content_html": "

Comments

", "url": "https://linus.schreibt.jetzt/posts/debugging-pid1.html", "title": "How to debug your initramfs init", "date_modified": "2024-02-20T09:50:18.000Z" }, { "content_html": "Comments", "url": "https://github.com/black7375/Firefox-UI-Fix/wiki/%5BArticle%5D-1.-How-to-make-a-better-default-Firefox-UI", "title": "How to make a better default Firefox UI", "date_modified": "2024-02-19T11:02:39.000Z" }, { "content_html": "Comments", "url": "https://taoofmac.com/space/blog/2024/02/03/2000", "title": "The Logitech G Cloud", "date_modified": "2024-02-19T09:18:20.000Z" }, { "content_html": "

Comments

", "url": "https://erikbern.com/2021/04/19/software-infrastructure-2.0-a-wishlist.html", "title": "Software infrastructure 2.0: a wishlist", "date_modified": "2024-02-18T20:27:24.000Z" }, { "content_html": "

Comments

", "url": "https://tempo.formkit.com/", "title": "Tempo", "date_modified": "2024-02-18T17:35:19.000Z" }, { "content_html": "Comments", "url": "https://www.mydbops.com/blog/postgresql-schema-changes-with-pg_osc/", "title": "Revolutionizing PostgreSQL Schema Changes with pg-osc", "date_modified": "2024-02-18T13:22:34.000Z" }, { "content_html": "Comments", "url": "https://git.kernel.dk/cgit/liburing/tree/examples/proxy.c", "title": "Basic proxy implementation using io_uring", "date_modified": "2024-02-17T23:46:17.000Z" }, { "content_html": "

Comments

", "url": "https://registerspill.thorstenball.com/p/from-1s-to-4ms", "title": "From 1s to 4ms", "date_modified": "2024-02-17T23:23:27.000Z" }, { "content_html": "Comments", "url": "https://www.defined.net/blog/nebula-is-not-the-fastest-mesh-vpn/", "title": "Nebula is not the fastest mesh VPN, but neither are the others", "date_modified": "2024-02-17T18:55:24.000Z" }, { "content_html": "Comments", "url": "https://github.com/shepherdjerred/macos-cross-compiler", "title": "Show HN: macOS-cross-compiler – Compile binaries for macOS on Linux", "date_modified": "2024-02-17T14:41:56.000Z" }, { "content_html": "

Comments

", "url": "http://www.chriswarbo.net/projects/nixos/nix_dependencies.html", "title": "Dependency solving in Nix", "date_modified": "2024-02-16T00:24:47.000Z" }, { "content_html": "Comments", "url": "https://pears.com/news/holepunch-unveils-groundbreaking-open-source-peer-to-peer-app-development-platform-pear-runtime/", "title": "Holepunch Unveils P2P Platform \"Pear Runtime\"", "date_modified": "2024-02-14T19:21:07.000Z" }, { "content_html": "Comments", "url": "https://console.dataherald.ai/playground", "title": "Show HN: Natural Language to SQL \"Text-to-SQL\" API", "date_modified": "2024-02-14T19:03:55.000Z" }, { "content_html": "Comments", "url": "https://www.ycombinator.com/rfs", "title": "YC: Requests for Startups", "date_modified": "2024-02-14T16:31:57.000Z" }, { "content_html": "Comments", "url": "https://www.rerun.io/blog/primary-query-caching", "title": "We sped up time series by 20-30x", "date_modified": "2024-02-14T14:03:37.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/fly-io-has-gpus-now/", "title": "Fly.io Has GPUs Now", "date_modified": "2024-02-13T22:06:51.000Z" }, { "content_html": "Comments", "url": "https://benhoyt.com/writings/flyio-and-tigris/", "title": "Switching from S3 to Tigris on Fly.io", "date_modified": "2024-02-13T18:22:04.000Z" }, { "content_html": "Comments", "url": "https://seb.jambor.dev/posts/systemd-by-example-part-1-minimization/", "title": "systemd by Example (2021)", "date_modified": "2024-02-12T11:48:51.000Z" }, { "content_html": "Comments", "url": "https://blog.stenmans.org/theBeamBook/", "title": "The Erlang Runtime System", "date_modified": "2024-02-12T07:24:43.000Z" }, { "content_html": "Comments", "url": "https://docs.google.com/spreadsheets/d/1OfRSXibZ2nIE9DGK6swwBZXgXwdCPKgp4SbPZwTexCg/htmlview", "title": "OpenZFS bug reports for native encryption", "date_modified": "2024-02-12T04:26:55.000Z" }, { "content_html": "Comments", "url": "https://yorickpeterse.com/articles/what-it-was-like-working-for-gitlab/", "title": "What it was like working for Gitlab", "date_modified": "2024-02-11T07:29:04.000Z" }, { "content_html": "Comments", "url": "https://getdeploying.com/reference/data-egress", "title": "How much 1 TB of egress costs by cloud provider", "date_modified": "2024-02-10T20:02:10.000Z" }, { "content_html": "Comments", "url": "https://github.com/honojs/hono/releases/tag/v4.0.0", "title": "Hono v4.0.0", "date_modified": "2024-02-09T11:55:10.000Z" }, { "content_html": "Comments", "url": "https://cep.dev/posts/every-infrastructure-decision-i-endorse-or-regret-after-4-years-running-infrastructure-at-a-startup/", "title": "Almost every infrastructure decision I endorse or regret", "date_modified": "2024-02-09T11:05:39.000Z" }, { "content_html": "Comments", "url": "https://github.com/awslabs/llrt", "title": "LLRT: A low-latency JavaScript runtime from AWS", "date_modified": "2024-02-08T16:46:06.000Z" }, { "content_html": "Comments", "url": "https://cyberus-technology.de/articles/vbox-kvm-public-release", "title": "VirtualBox KVM Public Release", "date_modified": "2024-02-08T10:20:25.000Z" }, { "content_html": "Comments", "url": "https://zenhorace.dev/blog/context-control-go/", "title": "Context Control in Go", "date_modified": "2024-02-08T08:40:58.000Z" }, { "content_html": "Comments", "url": "https://netflixtechblog.com/rebuilding-netflix-video-processing-pipeline-with-microservices-4e5e6310e359?gi=d48b3333df75", "title": "Rebuilding Netflix's video processing pipeline with microservices", "date_modified": "2024-02-08T08:22:56.000Z" }, { "content_html": "Comments", "url": "https://flawless.dev/essays/when-letting-it-crash-is-not-enough/", "title": "When \"letting it crash\" is not enough", "date_modified": "2024-02-08T05:07:57.000Z" }, { "content_html": "Comments", "url": "https://www.unison.cloud/", "title": "Unison Cloud", "date_modified": "2024-02-07T16:59:50.000Z" }, { "content_html": "Comments", "url": "https://ente.io/blog/tech/sqlite-objectbox-isar/", "title": "SQLite Isn't Enough", "date_modified": "2024-02-07T14:52:24.000Z" }, { "content_html": "Comments", "url": "https://www.luxcapital.com/securities/techcrunch-plus-termination", "title": "TechCrunch+ Termination", "date_modified": "2024-02-07T11:28:44.000Z" }, { "content_html": "Comments", "url": "https://pluralistic.net/2024/02/06/spoil-the-bunch/#dma", "title": "Apple to EU: \"Go fuck yourself\"", "date_modified": "2024-02-07T08:11:45.000Z" }, { "content_html": "Comments", "url": "https://depot.dev/blog/buildkit-in-depth", "title": "BuildKit in depth: Docker's build engine explained", "date_modified": "2024-02-06T17:31:24.000Z" }, { "content_html": "Comments", "url": "https://helixml.substack.com/p/how-we-got-fine-tuning-mistral-7b", "title": "How we got fine-tuning Mistral-7B to not suck", "date_modified": "2024-02-06T07:12:19.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/monadic/status/1754530336120140116", "title": "Weaveworks Is Shuting Down", "date_modified": "2024-02-05T16:51:18.000Z" }, { "content_html": "Comments", "url": "https://github.com/cfahlgren1/natural-sql", "title": "Show HN: Natural-SQL-7B, a strong text-to-SQL model", "date_modified": "2024-02-05T14:22:36.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/deno-in-2023", "title": "Deno in 2023", "date_modified": "2024-02-05T09:23:53.000Z" }, { "content_html": "Comments", "url": "https://folk.computer/", "title": "Folk Computer", "date_modified": "2024-02-03T15:52:40.000Z" }, { "content_html": "Comments", "url": "https://www.awwsmm.com/blog/make-invalid-states-unrepresentable", "title": "Make Invalid States Unrepresentable", "date_modified": "2024-02-01T20:49:16.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/macaroons-escalated-quickly/", "title": "Macaroons Escalated Quickly", "date_modified": "2024-01-31T14:39:19.000Z" }, { "content_html": "

Comments

", "url": "https://blog.cloudflare.com/introducing-foundations-our-open-source-rust-service-foundation-library", "title": "Introducing Foundations - our open source Rust service foundation library", "date_modified": "2024-01-31T13:39:29.000Z" }, { "content_html": "Comments", "url": "https://github.com/timeplus-io/proton", "title": "Proton, a fast and lightweight alternative to Apache Flink", "date_modified": "2024-01-30T22:42:36.000Z" }, { "content_html": "Comments", "url": "https://www.ubicloud.com/use-cases/github-actions", "title": "Show HN: Open-source x64 and Arm GitHub runners. Reduces GitHub Actions bill 10x", "date_modified": "2024-01-30T16:14:42.000Z" }, { "content_html": "Comments", "url": "https://www.ubicloud.com/use-cases/github-actions", "title": "Show HN: Open-source x64 and Arm GitHub runners", "date_modified": "2024-01-30T16:14:42.000Z" }, { "content_html": "Comments", "url": "https://sudhir.io/the-big-little-guide-to-message-queues", "title": "The Big Little Guide to Message Queues", "date_modified": "2024-01-29T19:08:52.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/AIatMeta/status/1752013879532782075", "title": "Meta AI releases Code Llama 70B", "date_modified": "2024-01-29T17:11:34.000Z" }, { "content_html": "Comments", "url": "https://www.todepond.com/wikiblogarden/better-computing/just/", "title": "Just", "date_modified": "2024-01-28T16:31:10.000Z" }, { "content_html": "Comments", "url": "https://missionlocal.org/2024/01/garry-tan-death-wish-sf-supervisors/", "title": "Garry Tan tech CEO and campaign donor wishes death upon SF politicians", "date_modified": "2024-01-28T04:12:13.000Z" }, { "content_html": "Comments", "url": "https://metebalci.com/blog/hello-ipv6/", "title": "Hello IPv6: a minimal tutorial for IPv4 users", "date_modified": "2024-01-27T20:09:19.000Z" }, { "content_html": "Comments", "url": "https://github.com/oasislinux/oasis", "title": "Oasis – a small, statically-linked Linux system", "date_modified": "2024-01-26T14:11:36.000Z" }, { "content_html": "Comments", "url": "https://blog.trailofbits.com/2024/01/25/we-build-x-509-chains-so-you-dont-have-to/", "title": "We build X.509 chains so you don't have to", "date_modified": "2024-01-25T14:20:28.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2024-01-23-good-devex-increases-productivity/", "title": "Good DevEx increases productivity. Here is the data", "date_modified": "2024-01-24T16:18:33.000Z" }, { "content_html": "Comments", "url": "https://www.infoq.com/articles/cloud-computing-post-serverless-trends/", "title": "Post-Serverless Era Trends", "date_modified": "2024-01-22T19:26:02.000Z" }, { "content_html": "Comments", "url": "https://www.jotaen.net/iE3XC/", "title": "CLI tool, written in Rust, to diff directory snapshots", "date_modified": "2024-01-22T19:14:30.000Z" }, { "content_html": "Comments", "url": "https://tuananh.net/2024/01/21/cloud-cost-optimization-at-scale-part-1/", "title": "Cutting down AWS cost by $150k per year simply by shutting things off", "date_modified": "2024-01-22T16:33:57.000Z" }, { "content_html": "Comments", "url": "https://supabase.com/blog/should-i-open-source-my-company", "title": "Should I Open Source my Company?", "date_modified": "2024-01-22T09:40:13.000Z" }, { "content_html": "Comments", "url": "https://bitbytebit.substack.com/p/the-size-of-your-backlog-is-inversely", "title": "The size of abacklog is inversely proportional to how often we talk to customers", "date_modified": "2024-01-21T19:18:00.000Z" }, { "content_html": "Comments", "url": "https://sourcehut.org/blog/2024-01-19-outage-post-mortem/", "title": "Sourcehut network outage post-mortem", "date_modified": "2024-01-19T15:55:40.000Z" }, { "content_html": "
\n
\n \"How\n

Enlarge (credit: Sam Rodriguez)

\n\n\n\n\n\n\n
\n

JUST OVER A DECADE AGO, Bitcoin appeared to many of its adherents to be the crypto-anarchist holy grail: truly private digital cash for the Internet.

\n

Satoshi Nakamoto, the cryptocurrency’s mysterious and unidentifiable inventor, had stated in an email introducing Bitcoin that “participants can be anonymous.” And the Silk Road dark-web drug market seemed like living proof of that potential, enabling the sale of hundreds of millions of dollars in illegal drugs and other contraband for bitcoin while flaunting its impunity from law enforcement.

\n

This is the story of the revelation in late 2013 that Bitcoin was, in fact, the opposite of untraceable—that its blockchain would actually allow researchers, tech companies, and law enforcement to trace and identify users with even more transparency than the existing financial system. That discovery would upend the world of cybercrime. Bitcoin tracing would, over the next few years, solve the mystery of the theft of a half-billion dollar stash of bitcoins from the world’s first crypto exchange, help enable the biggest dark-web drug market takedown in history, lead to the arrest of hundreds of pedophiles around the world in the bust of the dark web’s largest child sexual abuse video site, and result in the first-, second-, and third-biggest law enforcement monetary seizures in the history of the US Justice Department.

Read 65 remaining paragraphs | Comments

", "url": "https://arstechnica.com/?p=1996584", "title": "How a 27-year-old busted the myth of Bitcoin’s anonymity", "date_modified": "2024-01-18T11:00:06.000Z" }, { "content_html": "Comments", "url": "https://michael.stapelberg.ch/posts/2024-01-17-systemd-indefinite-service-restarts/", "title": "Systemd: Enable Indefinite Service Restarts", "date_modified": "2024-01-17T20:08:34.000Z" }, { "content_html": "Comments", "url": "https://lgug2z.com/articles/deploying-a-cloudflare-r2-backed-nix-binary-cache-attic-on-fly-io/", "title": "Cloudflare R2-Backed Nix Binary Cache on Fly.io", "date_modified": "2024-01-17T16:32:40.000Z" }, { "content_html": "Comments", "url": "https://read.engineerscodex.com/p/meta-xfaas-serverless-functions-explained", "title": "Meta's serverless platform processing trillions of function calls a day (2023)", "date_modified": "2024-01-17T15:57:03.000Z" }, { "content_html": "Comments", "url": "https://read.engineerscodex.com/p/how-apple-built-icloud-to-store-billions", "title": "Apple built iCloud to store billions of databases", "date_modified": "2024-01-17T15:05:44.000Z" }, { "content_html": "Comments", "url": "https://willowprotocol.org/", "title": "Willow Protocol", "date_modified": "2024-01-17T12:27:50.000Z" }, { "content_html": "Comments", "url": "https://status.kagi.com/issues/2024-01-12-kagi-down-on-some-regions/", "title": "Post-mortem for last week's incident at Kagi", "date_modified": "2024-01-16T21:04:39.000Z" }, { "content_html": "

Comments

", "url": "https://www.reddit.com/r/BSD/s/09ZWUI3Ufa", "title": "A NetBSD/amd64 guest can now boot in 40ms", "date_modified": "2024-01-16T20:42:48.000Z" }, { "content_html": "Comments", "url": "https://nixcademy.com/2024/01/15/nix-on-macos/", "title": "Going declarative on macOS with Nix and Nix-Darwin", "date_modified": "2024-01-15T19:10:34.000Z" }, { "content_html": "Comments", "url": "https://www.bitsand.cloud/posts/slashing-data-transfer-costs/", "title": "Slashing Data Transfer Costs in AWS by 99%", "date_modified": "2024-01-15T08:19:13.000Z" }, { "content_html": "Comments", "url": "https://nixos.org/", "title": "NixOS: Declarative Builds and Deployments", "date_modified": "2024-01-14T22:27:47.000Z" }, { "content_html": "Comments", "url": "https://www.grepular.com/Skiff_Emails_Various_Privacy_Failures", "title": "Skiff: Various Privacy Failures", "date_modified": "2024-01-14T19:40:35.000Z" }, { "content_html": "Comments", "url": "https://www.ypsidanger.com/announcing-project-bluefin/", "title": "Project Bluefin: an immutable, developer-focused, Cloud-native Linux", "date_modified": "2024-01-14T17:23:30.000Z" }, { "content_html": "Comments", "url": "https://devopscycle.com/blog/the-ultimate-docker-cheat-sheet/", "title": "The Ultimate Docker Cheat Sheet", "date_modified": "2024-01-14T09:45:24.000Z" }, { "content_html": "Comments", "url": "https://betterstack.com/community/guides/scaling-docker/podman-vs-docker/", "title": "Exploring Podman: A More Secure Docker Alternative", "date_modified": "2024-01-13T17:00:27.000Z" }, { "content_html": "Comments", "url": "https://github.com/aspen-cloud/triplit", "title": "Triplit: Open-source DB that syncs data between server and browser in real-time", "date_modified": "2024-01-13T05:13:46.000Z" }, { "content_html": "Comments", "url": "https://github.com/aws/aws-lc-rs", "title": "AWS Libcrypto for Rust", "date_modified": "2024-01-13T00:11:00.000Z" }, { "content_html": "

Comments

", "url": "https://betterstack.com/community/guides/scaling-docker/podman-vs-docker/", "title": "Exploring Podman: A More Secure Docker Alternative", "date_modified": "2024-01-12T11:28:36.000Z" }, { "content_html": "Comments", "url": "https://www.tweag.io/blog/2023-10-05-cli-ux-in-topiary/", "title": "CLI user experience case study", "date_modified": "2024-01-12T11:01:47.000Z" }, { "content_html": "Comments", "url": "https://outage.sr.ht/", "title": "Statement regarding the ongoing Sourcehut outage", "date_modified": "2024-01-12T09:43:52.000Z" }, { "content_html": "Comments", "url": "https://www.trevoragilbert.com/posts/a-love-letter-to-tinkerable-software/", "title": "A Love Letter to Tinkerable Software", "date_modified": "2024-01-11T23:42:25.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=38959668", "title": "Ask HN: How can I make local dev with containers hurt less?", "date_modified": "2024-01-11T21:48:29.000Z" }, { "content_html": "Comments", "url": "https://cloud.google.com/blog/products/networking/eliminating-data-transfer-fees-when-migrating-off-google-cloud/", "title": "Removing data transfer fees when moving off Google Cloud", "date_modified": "2024-01-11T15:49:27.000Z" }, { "content_html": "Comments", "url": "https://www.jonashietala.se/blog/2024/01/11/exploring_the_gleam_ffi/", "title": "Exploring the Gleam FFI", "date_modified": "2024-01-11T12:08:00.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/pgspider/dynamodb_fdw", "title": "DynamoDB Foreign Data Wrapper for PostgreSQL", "date_modified": "2024-01-11T04:48:40.000Z" }, { "content_html": "Comments", "url": "https://www.metahead.dev/", "title": "Metahead – An enterprise-grade, Git-based metarepo", "date_modified": "2024-01-10T21:27:28.000Z" }, { "content_html": "Comments", "url": "https://en.wikipedia.org/wiki/British_Post_Office_scandal", "title": "British Post Office Scandal", "date_modified": "2024-01-10T08:29:26.000Z" }, { "content_html": "Comments", "url": "https://www.nature.com/articles/d41586-024-00012-z", "title": "Ibogaine banishes PTSD, small study finds", "date_modified": "2024-01-09T22:15:57.000Z" }, { "content_html": "Comments", "url": "https://www.phoronix.com/news/Linux-6.8-Networking", "title": "Linux 6.8 Network Optimizations Can Boost TCP Performance by ~40%", "date_modified": "2024-01-09T21:32:18.000Z" }, { "content_html": "Comments", "url": "https://www.rabbit.tech/keynote", "title": "Rabbit: LLM-First Mobile Phone", "date_modified": "2024-01-09T18:44:17.000Z" }, { "content_html": "Comments", "url": "https://jade.fyi/blog/flakes-arent-real/", "title": "Flakes aren't real and cannot hurt you: using Nix flakes the non-flake way", "date_modified": "2024-01-09T18:05:56.000Z" }, { "content_html": "Comments", "url": "https://blog.bytebytego.com/p/how-discord-serves-15-million-users", "title": "Discord Serves 15M Users on One Server", "date_modified": "2024-01-09T17:12:31.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/josevalim/status/1744395345872683471", "title": "Today we celebrate by announcing that Elixir is a gradually typed language", "date_modified": "2024-01-08T16:29:43.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/josevalim/status/1744395345872683471", "title": "Elixir is now a gradually typed language", "date_modified": "2024-01-08T16:29:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/wagoodman/dive", "title": "Dive: A tool for exploring a Docker image, layer contents and more", "date_modified": "2024-01-08T15:35:38.000Z" }, { "content_html": "

Comments

", "url": "https://media.ccc.de/v/all-systems-go-2023-193-adventures-of-linux-userspace-at-meta", "title": "Adventures of Linux Userspace at Meta", "date_modified": "2024-01-08T00:57:42.000Z" }, { "content_html": "Comments", "url": "https://vitaut.net/posts/2024/faster-cpp-compile-times/", "title": "Optimizing the unoptimizable: a journey to faster C++ compile times", "date_modified": "2024-01-06T19:21:06.000Z" }, { "content_html": "Comments", "url": "https://sentry.io/legal/changelog/", "title": "Sentry new TOS to use data to train AI with no opt-out", "date_modified": "2024-01-05T08:48:58.000Z" }, { "content_html": "Comments", "url": "https://github.com/binkley/modern-java-practices", "title": "Modern Java/JVM Build Practices", "date_modified": "2024-01-05T03:18:55.000Z" }, { "content_html": "Comments", "url": "https://commandcenter.blogspot.com/2024/01/what-we-got-right-what-we-got-wrong.html", "title": "Go: What we got right, what we got wrong", "date_modified": "2024-01-04T21:16:08.000Z" }, { "content_html": "Comments", "url": "https://blog.digger.dev/how-open-id-connect-works-illustrated/", "title": "How Open ID Connect Works", "date_modified": "2024-01-04T16:03:47.000Z" }, { "content_html": "

Comments

", "url": "https://bas.codes/posts/cloudbench2312", "title": "Performance Benchmarks of Cloud Virtual Machines", "date_modified": "2024-01-03T18:54:54.000Z" }, { "content_html": "Comments", "url": "https://github.com/google/gvisor", "title": "Gvisor: Application Kernel for Containers", "date_modified": "2024-01-03T03:50:29.000Z" }, { "content_html": "Comments", "url": "https://protocol.bryanjohnson.com/", "title": "Blueprint health protocol – Bryan Johnson (founder braintree/Venmo)", "date_modified": "2024-01-02T08:31:16.000Z" }, { "content_html": "Comments", "url": "https://protocol.bryanjohnson.com/", "title": "Blueprint health protocol", "date_modified": "2024-01-02T08:31:16.000Z" }, { "content_html": "Comments", "url": "https://thisisimportant.net/posts/content-as-a-graph/", "title": "Displaying Content as a Graph", "date_modified": "2024-01-01T19:52:04.000Z" }, { "content_html": "Comments", "url": "https://www.bloomberg.com/news/articles/2023-12-30/ant-completes-process-of-removing-billionaire-jack-ma-s-control", "title": "Ant Completes Process of Removing Jack Ma's Control", "date_modified": "2024-01-01T17:23:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/ublue-os/bazzite", "title": "Bazzite – a SteamOS-like OCI image for desktop, living room, and handheld PCs", "date_modified": "2023-12-31T22:21:56.000Z" }, { "content_html": "Comments", "url": "https://matklad.github.io/2023/12/31/O(1)-build-file.html#O-1-Build-File", "title": "O(1) Build File", "date_modified": "2023-12-31T21:17:44.000Z" }, { "content_html": "Comments", "url": "http://devops-pipeline.com/", "title": "Mazzle – A Pipelines as Code Tool", "date_modified": "2023-12-31T11:15:08.000Z" }, { "content_html": "Comments", "url": "http://devops-pipeline.com/", "title": "Mazzle – A pipelines as code tool", "date_modified": "2023-12-31T11:15:08.000Z" }, { "content_html": "Comments", "url": "https://techbooks.substack.com/p/why-asking-your-customers-what-they", "title": "Asking your customers what they want doesn't work", "date_modified": "2023-12-30T10:30:33.000Z" }, { "content_html": "Comments", "url": "https://incident.io/blog/festive-macbooks", "title": "Tracking developer build times to decide if the M3 MacBook is worth upgrading", "date_modified": "2023-12-29T19:57:58.000Z" }, { "content_html": "Comments", "url": "https://pythonspeed.com/articles/alpine-docker-python/", "title": "Using Alpine can make Python Docker builds 50× slower", "date_modified": "2023-12-28T20:31:14.000Z" }, { "content_html": "Comments", "url": "https://loderunnerwebgame.com/game/", "title": "Lode Runner (HTML5 Remake)", "date_modified": "2023-12-28T12:24:06.000Z" }, { "content_html": "Comments", "url": "https://media.ccc.de/b/congress/2023", "title": "37C3 Video List", "date_modified": "2023-12-28T10:03:56.000Z" }, { "content_html": "Comments", "url": "https://media.ccc.de/v/37c3-12326-you_ve_just_been_fucked_by_psyops", "title": "You've just been fucked by psyops [video]", "date_modified": "2023-12-28T09:17:46.000Z" }, { "content_html": "Comments", "url": "https://github.com/macos-fuse-t/fuse-t", "title": "FUSE-T is a kext-less implementation of FUSE for macOS that uses NFSv4", "date_modified": "2023-12-26T20:13:48.000Z" }, { "content_html": "Comments", "url": "https://github.com/jstrieb/just.sh", "title": "Show HN: Just.sh – compiler that turns Justfiles into portable shell scripts", "date_modified": "2023-12-26T14:30:05.000Z" }, { "content_html": "Comments", "url": "https://obsidian.md/changelog/2023-12-26-desktop-v1.5.3/", "title": "Obsidian 1.5 Desktop (Public)", "date_modified": "2023-12-26T13:24:41.000Z" }, { "content_html": "Comments", "url": "https://lucumr.pocoo.org/2023/12/25/life-and-death-of-open-source/", "title": "The life and death of open source companies", "date_modified": "2023-12-26T04:45:59.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=38755180", "title": "If only someone told me this before my first startup", "date_modified": "2023-12-24T17:54:36.000Z" }, { "content_html": "Comments", "url": "https://www.vulture.com/2023/12/hbomberguy-interview-james-somerton-plagiarism.html", "title": "Hbomberguy didn't want to make that 4-hour plagiarism video", "date_modified": "2023-12-23T16:56:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/maypok86/otter", "title": "Otter, Fastest Go in-memory cache based on S3-FIFO algorithm", "date_modified": "2023-12-23T15:49:21.000Z" }, { "content_html": "Comments", "url": "https://hachyderm.io/@paulbiggar/111627367674590120", "title": "Paul Biggar removed from CircleCI board for pro-Palestine blog post", "date_modified": "2023-12-23T13:18:50.000Z" }, { "content_html": "Comments", "url": "https://read.engineerscodex.com/p/how-pinterest-scaled-to-11-million", "title": "How Pinterest scaled", "date_modified": "2023-12-23T08:51:55.000Z" }, { "content_html": "Comments", "url": "https://www.tokyodev.com/articles/obtaining-a-business-manager-visa-in-japan", "title": "How I Obtained a Business Manager Visa in Japan", "date_modified": "2023-12-22T11:20:28.000Z" }, { "content_html": "Comments", "url": "https://fusionauth.io/articles/identity-basics/what-is-oidc", "title": "What Is OIDC?", "date_modified": "2023-12-22T01:20:12.000Z" }, { "content_html": "Comments", "url": "https://www.builder.io/blog/structured-clone", "title": "Deep Cloning Objects in JavaScript, the Modern Way", "date_modified": "2023-12-21T23:13:25.000Z" }, { "content_html": "Comments", "url": "https://medium.com/@prajzler/webassembly-orchestration-with-visual-flows-eef4df56bde2", "title": "WASM Container Orchestration with Visual Flows", "date_modified": "2023-12-21T15:34:08.000Z" }, { "content_html": "Comments", "url": "https://isovalent.com/blog/post/cisco-acquires-isovalent/", "title": "Cisco to acquire Isovalent", "date_modified": "2023-12-21T14:15:06.000Z" }, { "content_html": "Comments", "url": "https://who.ldelossa.is/posts/ebpf-networking-technique-packet-redirection/", "title": "eBPF Networking Techniques – Packet Redirection", "date_modified": "2023-12-21T13:18:30.000Z" }, { "content_html": "Comments", "url": "https://www.panoptica.app/research/7-ways-to-escape-a-container", "title": "How to Escape a Container", "date_modified": "2023-12-20T22:40:14.000Z" }, { "content_html": "Comments", "url": "https://www.zachleat.com/web/netlify-and-nextjs/", "title": "Netlify's disingenuous survey-based attack on Next.js (and eleventy, too)", "date_modified": "2023-12-20T14:04:05.000Z" }, { "content_html": "Comments", "url": "https://antonz.org/trying-chdb/", "title": "Trying chDB, an embeddable ClickHouse engine", "date_modified": "2023-12-20T13:31:30.000Z" }, { "content_html": "Comments", "url": "https://github.com/wavetermdev/waveterm", "title": "Show HN: Wave – Modern Open-Source Terminal (macOS and Linux)", "date_modified": "2023-12-19T21:29:32.000Z" }, { "content_html": "Comments", "url": "https://x.com/dominicstpierre/status/1737069029976617420?s=46", "title": "Did the cloud made us over-engineer some systems that could have been simpler?", "date_modified": "2023-12-19T20:07:16.000Z" }, { "content_html": "Comments", "url": "https://jepsen.io/analyses/mysql-8.0.34", "title": "Jepsen: MySQL 8.0.34", "date_modified": "2023-12-19T14:17:25.000Z" }, { "content_html": "Comments", "url": "https://thebsdbox.co.uk/2023/12/08/Application-traffic-with-eBPF/", "title": "Application Traffic with eBPF", "date_modified": "2023-12-19T13:48:11.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/fks/", "title": "Fly Kubernetes", "date_modified": "2023-12-18T17:21:34.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=PeKMEXUrlq4", "title": "Behind the scenes scaling ChatGPT and the OpenAI APIs [video] - Eng Mgr @ OpenAI", "date_modified": "2023-12-18T12:22:36.000Z" }, { "content_html": "

Comments

", "url": "https://lu.sagebl.eu/notes/maybe-we-dont-need-uuidv7/", "title": "Maybe We Don’t Need UUIDv7 After All", "date_modified": "2023-12-17T11:37:41.000Z" }, { "content_html": "Comments", "url": "https://jack-vanlightly.com/blog/2023/11/29/s3-express-one-zone-not-quite-what-i-hoped-for", "title": "S3 Express One Zone, not quite what I hoped for", "date_modified": "2023-12-16T21:29:09.000Z" }, { "content_html": "Comments", "url": "https://github.com/francoismichel/ssh3", "title": "SSH3: SSHv2 using HTTP/3 and QUIC", "date_modified": "2023-12-16T15:06:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/wesql/wescale", "title": "Show HN: I've built a MySQL proxy that supports online DDL", "date_modified": "2023-12-16T05:03:34.000Z" }, { "content_html": "Comments", "url": "https://blaz.is/blog/post/mfio-release/", "title": "Mfio – Completion I/O for Rust", "date_modified": "2023-12-15T18:06:04.000Z" }, { "content_html": "Comments", "url": "https://changelog.com/posts/nixos-fatal-flaw", "title": "NixOS has one fatal flaw", "date_modified": "2023-12-15T17:31:13.000Z" }, { "content_html": "Comments", "url": "https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-seconds", "title": "We clone a running VM in 2 seconds", "date_modified": "2023-12-15T07:13:47.000Z" }, { "content_html": "Comments", "url": "https://www.hashicorp.com/blog/mitchell-reflects-as-he-departs-hashicorp", "title": "Mitchell reflects as he departs HashiCorp", "date_modified": "2023-12-14T21:27:17.000Z" }, { "content_html": "Comments", "url": "https://community.ui.com/questions/Security-Issue-Cloud-Site-Manager-presented-me-your-consoles-not-mine/376ec514-572d-476d-b089-030c4313888c", "title": "Ubiquiti showing other users' consoles", "date_modified": "2023-12-14T16:42:06.000Z" }, { "content_html": "Comments", "url": "https://www.biscuitsec.org/", "title": "Biscuit Authorization", "date_modified": "2023-12-13T23:18:57.000Z" }, { "content_html": "Comments", "url": "https://ubuntu.com/blog/ubuntu-performance-engineering-with-frame-pointers-by-default", "title": "Ubuntu 24.04 LTS will enable frame pointers by default", "date_modified": "2023-12-13T16:23:01.000Z" }, { "content_html": "Comments", "url": "https://supabase.com/blog/edge-functions-node-npm", "title": "Edge Functions: Node and native NPM compatibility", "date_modified": "2023-12-12T18:59:38.000Z" }, { "content_html": "

Comments

", "url": "https://antonz.org/trying-chdb/", "title": "Trying chDB, an embeddable ClickHouse engine", "date_modified": "2023-12-12T12:39:05.000Z" }, { "content_html": "Comments", "url": "https://www.theverge.com/23945184/epic-v-google-fortnite-play-store-antitrust-trial-updates", "title": "Epic vs. Google: Google Loses", "date_modified": "2023-12-12T00:21:07.000Z" }, { "content_html": "Comments", "url": "https://www.pcgamer.com/for-dooms-30th-anniversary-the-johns-romero-and-carmack-reunited-to-celebrate-the-fps-that-changed-everything-i-want-to-thank-everybody-in-the-doom-community-for-keeping-this-game-alive/", "title": "John Carmack and John Romero reunited to talk DOOM on its 30th Anniversary", "date_modified": "2023-12-11T01:14:13.000Z" }, { "content_html": "Comments", "url": "https://trippy.cli.rs/", "title": "Trippy – A Network Diagnostic Tool", "date_modified": "2023-12-10T03:46:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/philips-labs/terraform-aws-github-runner", "title": "Terraform module for scalable GitHub action runners on AWS", "date_modified": "2023-12-09T05:03:02.000Z" }, { "content_html": "Comments", "url": "https://www.theregister.com/2023/12/08/hashicorp_openbao_fork/", "title": "HashiCorp Vault Forked into OpenBao", "date_modified": "2023-12-09T03:27:10.000Z" }, { "content_html": "Comments", "url": "https://www.warpbuild.com/", "title": "Show HN: WarpBuild – x86-64 and arm GitHub Action runners for 30% faster builds", "date_modified": "2023-12-08T16:11:20.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/choose-your-ip/", "title": "Choose your own IP", "date_modified": "2023-12-07T17:29:50.000Z" }, { "content_html": "Comments", "url": "https://spotlightjs.com/", "title": "Spotlight: Sentry for Development", "date_modified": "2023-12-06T17:03:27.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/rethinking-serverless-with-flame/", "title": "Rethinking Serverless with Flame", "date_modified": "2023-12-06T12:03:39.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/privatevoid-net/nix-super", "title": "Nix Super", "date_modified": "2023-12-05T18:15:13.000Z" }, { "content_html": "Comments", "url": "https://practicalbetterments.com/switch-off-bad-tv-settings/", "title": "Switch off bad TV settings", "date_modified": "2023-12-04T16:08:32.000Z" }, { "content_html": "Comments", "url": "https://blog.carlosgaldino.com/writing-a-file-system-from-scratch-in-rust.html", "title": "Writing a file system from scratch in Rust", "date_modified": "2023-12-04T00:20:22.000Z" }, { "content_html": "Comments", "url": "https://blog.trailofbits.com/2023/11/06/adding-build-provenance-to-homebrew/", "title": "Adding Build Provenance to Homebrew", "date_modified": "2023-12-03T02:16:55.000Z" }, { "content_html": "Comments", "url": "https://rachelbythebay.com/w/2023/11/30/armv6/", "title": "Clang now makes binaries an original Pi B+ can't run", "date_modified": "2023-12-03T02:07:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/infinyon/fluvio", "title": "Show HN: Fluvio – Distributed stream processing system written in Rust and WASM", "date_modified": "2023-12-03T00:35:38.000Z" }, { "content_html": "Comments", "url": "https://github.com/json-web-proofs/json-web-proofs", "title": "JSON Web Proofs", "date_modified": "2023-12-02T22:36:27.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=38499134", "title": "Ask HN: What are some unpopular technologies you wish people knew more about?", "date_modified": "2023-12-02T15:16:38.000Z" }, { "content_html": "Comments", "url": "https://github.com/zitadel/oidc", "title": "Easy to use OpenID Connect client and server library written for Go", "date_modified": "2023-12-02T05:48:10.000Z" }, { "content_html": "Comments", "url": "https://blog.railway.app/p/gcp-incidents", "title": "GCP Incidents", "date_modified": "2023-12-02T02:14:51.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/JustJake/status/1730644510592692627?s=20", "title": "I never got a response after Google assured me that they'd do a full retro", "date_modified": "2023-12-01T19:03:28.000Z" }, { "content_html": "Comments", "url": "https://sqlsync.dev/posts/stop-building-databases/", "title": "SQLSync – Stop Building Databases", "date_modified": "2023-12-01T17:19:28.000Z" }, { "content_html": "Comments", "url": "https://noiselith.com/", "title": "Easy Stable Diffusion XL in your device, offline", "date_modified": "2023-12-01T14:34:50.000Z" }, { "content_html": "

Comments

", "url": "https://xata.io/blog/pgroll-internals", "title": "How pgroll works under the hood", "date_modified": "2023-11-30T19:44:13.000Z" }, { "content_html": "Comments", "url": "https://www.phoronix.com/forums/forum/phoronix/latest-phoronix-articles/1424461-openzfs-is-still-battling-a-data-corruption-issue?p=1424496#post1424496", "title": "In OpenZFS and Btrfs, everyone was just guessing", "date_modified": "2023-11-30T11:13:51.000Z" }, { "content_html": "

Today we are excited to announce the general availability of SaaS Quick Launch, a new feature in AWS Marketplace that makes it easy and secure to deploy SaaS products.

\n

Before SaaS Quick Launch, configuring and launching third-party SaaS products could be time-consuming and costly, especially in certain categories like security and monitoring. Some products require hours of engineering time to manually set up permissions policies and cloud infrastructure. Manual multistep configuration processes also introduce risks when buyers rely on unvetted deployment templates and instructions from third-party resources.

\n

SaaS Quick Launch helps buyers make the deployment process easy, fast, and secure by offering step-by-step instructions and resource deployment using preconfigured AWS CloudFormation templates. The software vendor and AWS validate these templates to ensure that the configuration adheres to the latest AWS security standards.

\n

Getting started with SaaS Quick Launch
It’s easy to find which SaaS products have Quick Launch enabled when you are browsing in AWS Marketplace. Products that have this feature configured have a Quick Launch tag in their description.

\n

\"Quick

\n

After completing the purchase process for a Quick Launch–enabled product, you will see a button to set up your account. That button will take you to the Configure and launch page, where you can complete the registration to set up your SaaS account, deploy any required AWS resources, and launch the SaaS product.

\n

\"Step

\n

The first step ensures that your account has the required AWS permissions to configure the software.

\n

\"Step

\n

The second step involves configuring the vendor account, either to sign in to an existing account or to create a new account on the vendor website. After signing in, the vendor site may pass essential keys and parameters that are needed in the next step to configure the integration.

\n

\"Step

\n

The third step allows you to configure the software and AWS integration. In this step, the vendor provides one or more CloudFormation templates that provision the required AWS resources to configure and use the product.

\n

\"Step

\n

The final step is to launch the software once everything is configured.

\n

\"Step

\n

Availability
Sellers can enable this feature in their SaaS product. If you are a seller and want to learn how to set this up in your product, check the Seller Guide for detailed instructions.

\n

To learn more about SaaS in AWS Marketplace, visit the service page and view all the available SaaS products currently in AWS Marketplace.

\n

— Marcia

", "url": "https://aws.amazon.com/blogs/aws/easily-deploy-saas-products-with-new-quick-launch-in-aws-marketplace/", "title": "Easily deploy SaaS products with new Quick Launch in AWS Marketplace", "date_modified": "2023-11-30T00:05:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/Mozilla-Ocho/llamafile", "title": "Llamafile lets you distribute and run LLMs with a single file", "date_modified": "2023-11-29T19:29:49.000Z" }, { "content_html": "Comments", "url": "https://github.com/reddit/redditsans", "title": "Reddit Sans", "date_modified": "2023-11-29T19:22:28.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/cron", "title": "Deno Cron", "date_modified": "2023-11-29T16:03:10.000Z" }, { "content_html": "Comments", "url": "https://sourcegraph.com/blog/ex-googler-guide-dev-tools", "title": "An ex-Googler's guide to dev tools (2020)", "date_modified": "2023-11-29T04:16:08.000Z" }, { "content_html": "Comments", "url": "https://www.darlinghq.org/", "title": "Darling: Run macOS Software on Linux", "date_modified": "2023-11-26T17:52:46.000Z" }, { "content_html": "Comments", "url": "https://computer.rip/2023-11-25-the-curse-of-docker.html", "title": "The Curse of Docker", "date_modified": "2023-11-26T01:14:03.000Z" }, { "content_html": "Comments", "url": "https://discourse.nixos.org/t/oci-images-is-there-something-similar-to-distroless-images-built-with-nix/35957", "title": "Fat OCI images are a cultural problem", "date_modified": "2023-11-25T01:56:45.000Z" }, { "content_html": "

Comments

", "url": "https://www.youtube.com/watch?v=HrtX0JrjekE", "title": "Zero-Downtime Live Migration of Stateful VMs on Kubernetes", "date_modified": "2023-11-24T22:36:52.000Z" }, { "content_html": "

Comments

", "url": "https://jmtd.net/log/dockerfile_arg_footgun/", "title": "Dockerfile ARG footgun", "date_modified": "2023-11-24T17:51:02.000Z" }, { "content_html": "

Comments

", "url": "https://fastify-vite.dev/", "title": "@fastify/vite: Titans Combined", "date_modified": "2023-11-24T16:00:39.000Z" }, { "content_html": "Comments", "url": "https://mastodon.social/@tdp_org/111462161186925649", "title": "BBC Outage", "date_modified": "2023-11-24T00:05:48.000Z" }, { "content_html": "Comments", "url": "https://blog.abctaylor.com/how-i-discovered-caching-cdns-were-throttling-my-everyday-browsing/", "title": "I discovered caching CDNs were throttling my everyday browsing", "date_modified": "2023-11-23T12:15:09.000Z" }, { "content_html": "Comments", "url": "https://blog.matiaspan.dev/posts/exploring-dagger-streamlining-ci-cd-pipelines-with-code/", "title": "Streamlining CI/CD Pipelines with Code", "date_modified": "2023-11-21T18:41:31.000Z" }, { "content_html": "Comments", "url": "https://tmpout.sh/3/20.html", "title": "Easylkb: Easy Linux Kernel Builder", "date_modified": "2023-11-21T12:42:20.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/satyanadella/status/1726509045803336122", "title": "Sam Altman, Greg Brockman and others to join Microsoft", "date_modified": "2023-11-20T07:56:13.000Z" }, { "content_html": "Comments", "url": "https://github.com/kamranahmedse/local-ses", "title": "Trap and test AWS SES emails locally", "date_modified": "2023-11-20T01:38:46.000Z" }, { "content_html": "Comments", "url": "https://github.com/rsms/inter/releases/tag/v4.0", "title": "The Inter font family version 4.0", "date_modified": "2023-11-20T01:04:47.000Z" }, { "content_html": "Comments", "url": "https://shavingtheyak.com/2023/10/28/hashicorps-terraform-cloud-rum-pricing-sticker-shock/", "title": "Terraform Cloud Pricing Changes Sticker Shock", "date_modified": "2023-11-19T16:23:27.000Z" }, { "content_html": "Comments", "url": "https://berkeleygraphics.com/typefaces/berkeley-mono/", "title": "Berkeley Mono Typeface", "date_modified": "2023-11-18T18:32:03.000Z" }, { "content_html": "Comments", "url": "https://etcha.dev", "title": "Show HN: Etcha – Infinite scale, serverless config management", "date_modified": "2023-11-18T15:39:55.000Z" }, { "content_html": "Comments", "url": "https://www.uber.com/en-NL/blog/nilaway-practical-nil-panic-detection-for-go/", "title": "Practical nil panic detection for Go", "date_modified": "2023-11-17T06:59:28.000Z" }, { "content_html": "Comments", "url": "https://pganalyze.com/blog/automatic-indexing-system-postgres-pganalyze-indexing-engine", "title": "An automatic indexing system for Postgres", "date_modified": "2023-11-17T06:38:18.000Z" }, { "content_html": "

Comments

", "url": "https://boinkor.net/2023/11/neat-github-actions-patterns-for-github-merge-queues/", "title": "Neat GitHub Actions patterns for GitHub Merge Queues", "date_modified": "2023-11-17T00:35:00.000Z" }, { "content_html": "Comments", "url": "https://linear.app/blog/planning-for-unplanned-work", "title": "Planning for Unplanned Work in Linear", "date_modified": "2023-11-16T19:58:29.000Z" }, { "content_html": "Comments", "url": "https://blog.oneuptime.com/moving-from-aws-to-bare-metal/", "title": "Moving from AWS to Bare-Metal saved us $230k per year", "date_modified": "2023-11-16T19:54:59.000Z" }, { "content_html": "Comments", "url": "https://blog.artie.so/best-practices-on-running-redshift-at-scale", "title": "Running Redshift at Scale", "date_modified": "2023-11-15T18:48:10.000Z" }, { "content_html": "Comments", "url": "https://inko-lang.org", "title": "Inko Programming Language", "date_modified": "2023-11-14T21:47:13.000Z" }, { "content_html": "Comments", "url": "https://github.com/eunomia-bpf/bpftime", "title": "Bpftime: Userspace eBPF runtime for fast Uprobe and Syscall hook and Plugins", "date_modified": "2023-11-14T20:10:36.000Z" }, { "content_html": "Comments", "url": "https://www.macchaffee.com/blog/2023/solarwinds-hack-lessons-learned/", "title": "We've learned nothing from the SolarWinds hack", "date_modified": "2023-11-13T21:56:51.000Z" }, { "content_html": "Comments", "url": "https://felix-knorr.net/posts/2023-11-11-github-actions.html", "title": "GitHub Actions Are a Problem", "date_modified": "2023-11-12T14:22:19.000Z" }, { "content_html": "Comments", "url": "https://insights.adadot.com/2023/11/10/serverless-at-scale-lessons-from-200-million-lambda-invocations/", "title": "Serverless at Scale: Lessons from 200M Lambda Invocations", "date_modified": "2023-11-11T03:39:59.000Z" }, { "content_html": "Comments", "url": "https://alan.norbauer.com/articles/browser-debugging-tricks", "title": "Weird debugging tricks the browser doesn’t want you to know", "date_modified": "2023-11-11T01:35:17.000Z" }, { "content_html": "Comments", "url": "https://www.system-transparency.org/", "title": "System Transparency: a security architecture for bare-metal servers", "date_modified": "2023-11-10T22:59:54.000Z" }, { "content_html": "Comments", "url": "https://github.com/githubnext/monaspace/blob/main/docs/Texture%20Healing.md", "title": "Texture Healing for Monospace Fonts", "date_modified": "2023-11-10T17:07:31.000Z" }, { "content_html": "Comments", "url": "https://xeiaso.net/notes/cursorless-alien-magic/", "title": "Cursorless is alien magic from the future", "date_modified": "2023-11-10T04:04:10.000Z" }, { "content_html": "Comments", "url": "https://monaspace.githubnext.com/", "title": "Monaspace", "date_modified": "2023-11-09T20:16:34.000Z" }, { "content_html": "Comments", "url": "https://aftermath.site/welcome-to-aftermath", "title": "Ex-Kotaku staff go independent and launch Aftermath", "date_modified": "2023-11-07T12:33:03.000Z" }, { "content_html": "Comments", "url": "https://www.wirehub.org/", "title": "Show HN: WireHub – easily create and share WireGuard networks", "date_modified": "2023-11-05T20:54:33.000Z" }, { "content_html": "Comments", "url": "https://github.com/ekzhang/sshx", "title": "Show HN: Sshx, a web-based collaborative terminal", "date_modified": "2023-11-05T15:44:23.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=38142782", "title": "Ask HN: How would French police locate suspects by tapping their devices?", "date_modified": "2023-11-04T16:52:46.000Z" }, { "content_html": "Comments", "url": "https://www.fermyon.com/blog/introducing-spin-v2", "title": "Spin 2.0 – open-source tool for building and running WASM apps", "date_modified": "2023-11-04T12:01:50.000Z" }, { "content_html": "Comments", "url": "https://charm.sh/blog/the-next-generation/", "title": "Charm has raised $6M in funding", "date_modified": "2023-11-03T08:46:30.000Z" }, { "content_html": "Comments", "url": "https://dansdatathoughts.substack.com/p/from-s3-to-r2-an-economic-opportunity", "title": "From S3 to R2: An economic opportunity", "date_modified": "2023-11-02T19:15:45.000Z" }, { "content_html": "Comments", "url": "https://cra.mr/sentry-from-the-beginning/", "title": "Sentry: From the Beginning", "date_modified": "2023-11-01T11:35:31.000Z" }, { "content_html": "Comments", "url": "https://github.com/streamdal/streamdal", "title": "Show HN: Streamdal – an open-source tail -f for your data", "date_modified": "2023-10-31T15:31:33.000Z" }, { "content_html": "Comments", "url": "https://rehanvdm.com/blog/should-you-use-a-lambda-monolith-lambdalith-for-the-api", "title": "Should you use a Lambda Monolith, a.k.a. Lambdalith, for your API?", "date_modified": "2023-10-31T09:29:41.000Z" }, { "content_html": "Comments", "url": "https://www.openstatus.dev/blog/migration-backend-from-vercel-to-fly", "title": "Migrating our backend from Vercel to Fly.io", "date_modified": "2023-10-29T20:50:26.000Z" }, { "content_html": "Comments", "url": "https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756", "title": "NixOS Reproducible Builds: minimal ISO successfully independently rebuilt", "date_modified": "2023-10-29T11:41:05.000Z" }, { "content_html": "Comments", "url": "https://boards.greenhouse.io/supabase/jobs/5005843004", "title": "Supabase (YC S20) Is Hiring a Technical Product Marketer (Fully Remote)", "date_modified": "2023-10-29T07:01:40.000Z" }, { "content_html": "Comments", "url": "https://github.com/CycodeLabs/raven", "title": "Raven – CI/CD Security Analyzer", "date_modified": "2023-10-29T05:23:57.000Z" }, { "content_html": "Comments", "url": "https://fly.io/phoenix-files/elixir-and-phoenix-can-do-it-all/", "title": "Elixir and Phoenix can do it all", "date_modified": "2023-10-28T19:57:16.000Z" }, { "content_html": "Comments", "url": "https://www.theguardian.com/media/2023/oct/27/us-asks-qatar-to-turn-down-the-volume-of-al-jazeera-news-coverage", "title": "US asks Qatar to 'turn down the volume' of Al Jazeera news coverage", "date_modified": "2023-10-27T21:15:55.000Z" }, { "content_html": "Comments", "url": "https://pakikiproxy.com/", "title": "Show HN: Pākiki Proxy – An intercepting proxy for penetration testing", "date_modified": "2023-10-27T15:35:49.000Z" }, { "content_html": "Comments", "url": "https://fusionauth.io/blog/grammarly-proves-ciam-not-optional", "title": "Grammarly's OAuth Mistakes", "date_modified": "2023-10-27T15:31:33.000Z" }, { "content_html": "Comments", "url": "https://nextjs.org/blog/next-14", "title": "Next.js 14", "date_modified": "2023-10-26T17:02:09.000Z" }, { "content_html": "Comments", "url": "https://oxide.computer/blog/the-cloud-computer", "title": "Oxide: The Cloud Computer", "date_modified": "2023-10-26T10:43:59.000Z" }, { "content_html": "Comments", "url": "https://www.epicweb.dev/why-i-wont-use-nextjs", "title": "I Won't Use Next.js", "date_modified": "2023-10-25T20:52:10.000Z" }, { "content_html": "Comments", "url": "https://nandovillalba.medium.com/why-i-think-gcp-is-better-than-aws-ea78f9975bda", "title": "I think GCP is better than AWS (2020)", "date_modified": "2023-10-25T19:01:54.000Z" }, { "content_html": "Comments", "url": "https://github.com/orbitalapi/orbital", "title": "Show HN: Orbital – Dynamically unifying APIs and data with no glue code", "date_modified": "2023-10-25T12:59:39.000Z" }, { "content_html": "Comments", "url": "https://flawless.dev/", "title": "Flawless – Durable execution engine for Rust", "date_modified": "2023-10-25T07:39:11.000Z" }, { "content_html": "

The AWS European Sovereign Cloud will allow government agencies, regulated industries, and the independent software vendors (ISVs) that support them to store sensitive data and run critical workloads on AWS infrastructure that is operated and supported by AWS employees located in and residents of the European Union (EU). The first Region will be located in Germany.

\n

Background
Late last year we announced the AWS Digital Sovereignty Pledge and made a commitment to offer you (and all AWS customers) the most advanced set of sovereignty controls and features available in the cloud. Since that announcement we have taken several important steps forward in fulfillment of that pledge:

\n

May 2023 – We announced that AWS Nitro System had been validated by an independent third-party to confirm that it contains no mechanism that allows anyone at AWS to access your data on AWS hosts. At the same time we announced that the AWS Key Management Service (KMS) External Key Store allows you to store keys outside of AWS and use them to encrypt data stored in AWS.

\n

August 2023 – We announced AWS Dedicated Local Zones, infrastructure that is fully managed by AWS and built for exclusive use by a customer or community, and placed in a customer-specified location or data center.

\n

AWS European Sovereign Cloud
The upcoming AWS European Sovereign Cloud will be separate from, and independent of, the eight existing AWS Regions already open in Frankfurt, Ireland, London, Milan, Paris, Stockholm, Spain, and Zurich. It will give you additional options for deployment, while providing AWS services, APIs, and tools that you are already familiar with. The design will help you meet your data residency, operational autonomy, and resiliency needs.

\n

In order to maintain separation between this cloud and the existing AWS Global Cloud you will need to create a fresh AWS account. The metadata you create such as data labels, categories, permissions, and configurations will be stored within the EU. This does not apply to AWS account information such as spend and billing data, which will be aggregated and used to ensure that you get favorable pricing within any applicable volume usage tiers.

\n

As I mentioned earlier, this cloud will be operated and supported by AWS employees located in and residents of the EU, with support available 24/7/365.

\n

The AWS European Sovereign Cloud will be operationally independent of the other regions, with separate in-Region billing and usage metering systems.

\n

Initial Region
The initial region will be located in Germany. It will launch with multiple Availability Zones, each in separate and distinct geographic locations, with enough distance between them to significantly reduce the risk of a single event impacting your business continuity. We will have additional details on the list of available services, instance types, and so forth as we get closer to the launch.

\n

Over time, this and other regions in this cloud will also function as parent regions for AWS Outposts and Dedicated Local Zones. These options give you even more flexibility with regard to isolation and in-country data residency. If you would like to express your interest in Dedicated Local Zones in your country, please contact your AWS account manager.

\n

Get Ready
You can start to build applications today in any of the existing regions and move them to the AWS European Sovereign Cloud when the region launches. You can also initiate conversations with your local regulatory authorities in order to better understand any issues that are specific to your particular location.

\n

— Jeff;

", "url": "https://aws.amazon.com/blogs/aws/in-the-works-aws-european-sovereign-cloud/", "title": "In the Works – AWS European Sovereign Cloud", "date_modified": "2023-10-25T05:06:17.000Z" }, { "content_html": "Comments", "url": "https://gittuf.github.io/", "title": "Gittuf – a security layer for Git", "date_modified": "2023-10-24T19:31:59.000Z" }, { "content_html": "Comments", "url": "https://www.jetbrains.com/writerside/", "title": "Writerside – a new technical writing environment from JetBrains", "date_modified": "2023-10-24T03:57:12.000Z" }, { "content_html": "Comments", "url": "https://arstechnica.com/security/2023/10/1password-detects-suspicious-activity-in-its-internal-okta-account/", "title": "1Password detects \"suspicious activity\" in its internal Okta account", "date_modified": "2023-10-24T00:55:56.000Z" }, { "content_html": "Comments", "url": "https://mswjs.io/blog/introducing-msw-2.0/", "title": "Introducing MSW 2.0", "date_modified": "2023-10-23T14:01:47.000Z" }, { "content_html": "Comments", "url": "https://mswjs.io/blog/introducing-msw-2.0/", "title": "MSW 2.0 – Mock Service Worker", "date_modified": "2023-10-23T14:01:47.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37985450", "title": "Ask HN: Was any Starfighter postmortem ever published?", "date_modified": "2023-10-23T13:39:47.000Z" }, { "content_html": "

Comments

", "url": "https://medium.com/snowflake/build-farm-visualizations-5a079477502d", "title": "Build farm visualizations", "date_modified": "2023-10-23T11:26:47.000Z" }, { "content_html": "Comments", "url": "https://eval.blog/research/microsoft-account-token-leaks-in-harvest/", "title": "Microsoft Account's OAuth tokens leaking via open redirect in Harvest", "date_modified": "2023-10-22T09:15:15.000Z" }, { "content_html": "Comments", "url": "https://old.reddit.com/r/hetzner/comments/17ccp3i/hetzner_does_run_a_mitm_proxy_in_front_of_my/", "title": "Hetzner does run a (MitM) proxy in front of my server", "date_modified": "2023-10-20T21:01:04.000Z" }, { "content_html": "Comments", "url": "https://www.devever.net/~hl/xmpp-incident", "title": "How to mitigate the Hetzner/Linode XMPP.ru MitM interception incident", "date_modified": "2023-10-20T20:31:29.000Z" }, { "content_html": "Comments", "url": "https://notes.valdikss.org.ru/jabber.ru-mitm/", "title": "Encrypted traffic interception on Hetzner and Linode targeting Jabber service", "date_modified": "2023-10-20T12:40:53.000Z" }, { "content_html": "

Comments

", "url": "https://blog.peerdb.io/using-temporal-to-scale-data-synchronization-at-peerdb", "title": "Using Temporal to Scale Data Synchronization at PeerDB", "date_modified": "2023-10-19T19:46:46.000Z" }, { "content_html": "Comments", "url": "https://fasterthanli.me/articles/just-paying-figma-15-dollars", "title": "Just paying Figma because nothing else works", "date_modified": "2023-10-19T17:58:01.000Z" }, { "content_html": "Comments", "url": "https://github.com/Exein-io/pulsar", "title": "Linux runtime security agent powered by eBPF", "date_modified": "2023-10-19T13:42:59.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37934488", "title": "Ask HN: Is anyone using Cloud Dev Environments (e.g. Codespaces/Replit) at work?", "date_modified": "2023-10-18T20:41:05.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37934488", "title": "Ask HN: Is anyone using cloud dev environments (e.g. Codespaces/Replit) at work?", "date_modified": "2023-10-18T20:41:05.000Z" }, { "content_html": "Comments", "url": "https://engineering.aviatrix.com/bpf-tailcalls-8ef4273747a5", "title": "BPF Tailcalls", "date_modified": "2023-10-18T19:23:41.000Z" }, { "content_html": "Comments", "url": "https://blog.jetbrains.com/blog/2023/10/16/ai-graphics-at-jetbrains-story/", "title": "AI Graphics at JetBrains", "date_modified": "2023-10-17T22:08:33.000Z" }, { "content_html": "Comments", "url": "https://github.com/TeamPiped/Piped", "title": "Piped – An alternative privacy-friendly YouTube front end", "date_modified": "2023-10-17T08:19:28.000Z" }, { "content_html": "Comments", "url": "https://www.prequel.co/blog/building-cross-cloud-identity-federation-in-go", "title": "Building cross-cloud identity federation in Go for secure data sharing", "date_modified": "2023-10-16T15:55:59.000Z" }, { "content_html": "Comments", "url": "https://invidious.io/", "title": "Invidious – An open source alternative front-end to YouTube", "date_modified": "2023-10-16T14:38:36.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37888728", "title": "Ask HN: What is your experience with Nano-Hydroxyapatite toothpaste?", "date_modified": "2023-10-15T11:01:25.000Z" }, { "content_html": "Comments", "url": "https://www.infoq.com/news/2023/10/cloudflare-sippy-migrate-s3/", "title": "Cloudflare Sippy: Incrementally Migrate Data from AWS S3 to Reduce Egress Fees", "date_modified": "2023-10-15T08:55:12.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/vercel_support/status/1713295474998772215", "title": "Vercel employee used customer information to pursue a personal trademark matter", "date_modified": "2023-10-15T05:10:49.000Z" }, { "content_html": "Comments", "url": "https://cloud.google.com/blog/products/databases/announcing-cloud-spanner-price-performance-updates", "title": "Google Cloud Spanner is now half the cost of Amazon DynamoDB", "date_modified": "2023-10-11T17:25:40.000Z" }, { "content_html": "Comments", "url": "https://buildkite.com/pricing", "title": "Buildkite has quietly removed their $9 \"Team\" plan", "date_modified": "2023-10-11T07:34:22.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37834908", "title": "Ask HN: SaaS Founders, What 3 advice would you give your younger selves?", "date_modified": "2023-10-10T17:26:37.000Z" }, { "content_html": "

Comments

", "url": "https://www.pulumi.com/blog/environments-secrets-configurations-management/", "title": "Introducing Pulumi ESC: Easy and Secure Environments, Secrets and Configuration", "date_modified": "2023-10-10T13:00:07.000Z" }, { "content_html": "Comments", "url": "https://tailscale.dev/blog/docker-mod-tailscale", "title": "The Tailscale Universal Docker Mod", "date_modified": "2023-10-08T16:51:28.000Z" }, { "content_html": "Comments", "url": "https://render.com/blog/knative", "title": "Scaling Knative to 100K+ Webapps", "date_modified": "2023-10-08T16:21:31.000Z" }, { "content_html": "Comments", "url": "https://discuss.linuxcontainers.org/t/incus-0-1-has-been-released/18036", "title": "Initial release of Incus, the LXD community fork", "date_modified": "2023-10-07T21:48:45.000Z" }, { "content_html": "Comments", "url": "https://ghiculescu.substack.com/p/11-years-of-saas-product-strategy", "title": "Looking back on SaaS product strategy", "date_modified": "2023-10-06T23:12:14.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37792300", "title": "Ask HN: Sales Tips for Solo Devs", "date_modified": "2023-10-06T15:44:33.000Z" }, { "content_html": "Comments", "url": "https://github.com/rails/rails/releases/tag/v7.1.0", "title": "Rails 7.1 Released", "date_modified": "2023-10-06T04:31:36.000Z" }, { "content_html": "Comments", "url": "https://blog.sigstore.dev/openpubkey-and-sigstore/", "title": "OpenPubKey and Sigstore", "date_modified": "2023-10-06T03:12:45.000Z" }, { "content_html": "

In the face of rapid digital transformation, a positive organizational culture and user-centric design are the backbone of successful software delivery. And while Artificial Intelligence (AI) is the center of so many contemporary technical conversations, the impact of AI development tools on teams is still in its infancy.

These are just some of the findings from the 2023 Accelerate State of DevOps Report, the annual report from Google Cloud’s DevOps Research and Assessment (DORA) team.

For nine years, the State of DevOps survey has assembled data from more than 36,000 professionals worldwide, making it the largest and longest-running research of its kind. This year, we took a deep dive into how high-performing DevOps performers bake these technical, process, and cultural capabilities into their development practices to drive success. Specifically, we explored three key outcomes of a having a DevOps practice and the capabilities that contribute to achieving them:

This year, we were working with a particularly robust data set: the total number of organic respondents increased by 3.6x compared to last year, allowing us to perform a deeper analysis of the relationship between ways of working and outcomes. Thank you to everyone who took the survey this year!

Measuring software delivery performance

Our research shows that an organization’s level of software delivery performance predicts overall performance, team performance, and employee well-being. In turn, we use the following measures to understand the throughput and stability of software changes:

Our analysis revealed four performance levels, including the return of the Elite performance level, which we did not detect in last year’s cohort. Elite performers around the world are able to achieve both throughput and stability.

\"2\"

Five key insights

There are several key takeaways for teams who want to understand how to improve their software delivery capabilities. Here are some of the key insights from this year’s report:

1. Establish a healthy culture

Culture is foundational to building technical capabilities, igniting technical performance, reaching organizational performance goals, and helping employees be successful. A healthy culture can help reduce burnout, increase productivity, and increase job satisfaction. Teams with generative cultures, composed of people who felt included and like they belonged on their team, have 30% higher organizational performance than organizations without a generative culture.

\"3\"

The aspects of culture that can improve employee well-being

2. Build with users in mind

Teams can deploy as fast and successfully as they'd like, but without the user in mind, it might be for naught. Our research shows that a user-centric approach to building applications and services is one of the strongest predictors of overall organizational performance. In fact, building with the user in mind appears to inform and drive improvements across all of the technical, process, and cultural capabilities we explore in the DORA research. Teams that focus on the user have 40% higher organizational performance than teams that don’t.

\"4\"

3. Amplify technical capabilities with quality documentation

High-quality documentation amplifies the impact that DevOps technical capabilities (for example, continuous integration and trunk-based development) have on organizational performance. This means that quality documentation not only helps establish these technical capabilities, but helps them matter. For example, SRE practices are estimated to have 1.4x more impact on organizational performance when high-quality documentation is in place. Overall, high-quality documentation leads to 25% higher team performance relative to low-quality documentation.

4. Distribute work fairly

People who identify as underrepresented and women or those who chose to self-describe their gender have higher levels of burnout. There are likely multiple systemic and environmental factors that cause this. Unsurprisingly, we find that respondents who take on more repetitive work are more likely to experience higher levels of burnout, and members of underrepresented groups are more likely to take on more repetitive work: Underrepresented respondents report 24% more burnout than those who are not underrepresented. Underrepresented respondents do 29% more repetitive work than those who are not underrepresented.And women or those who self-reported their gender do 40% more repetitive work than men.

5. Increase infrastructure flexibility with cloud

Teams can get the most value out of the cloud by leveraging the characteristics of cloud like rapid elasticity and on-demand self-service. These characteristics predict a more flexible infrastructure. Using a public cloud, for example, leads to a 22% increase in infrastructure flexibility relative to not using the cloud. This flexibility, in turn, leads to teams with 30% higher organizational performance than those with inflexible infrastructures.

AI: we're just getting started

There is a lot of enthusiasm about the potential of AI development tools. We saw this in this year’s results — in fact a majority of respondents are incorporating at least some AI into the tasks we included in our survey. But we anticipate that it will take some time for AI-powered tools to come into widespread and coordinated use in the industry. We are very interested in seeing how adoption grows over time and the impact that growth will have on performance measures and outcomes that are important to organizations. Here’s where we are seeing the adoption of AI tools today:

\"5\"

Applying insights from DORA in your context

The key takeaway from DORA’s research is that high performance requires continuous improvement. Regularly measure outcomes across your organization, teams, and employees. Identify areas for optimization and make incremental changes to dial up performance.

Don't let these insights sit on a shelf — put them into action. Contextualize the findings based on your team's current practices and pain points. Have open conversations about your bottlenecks. Comparing your metrics year-over-year is more meaningful than comparing yourself to other companies. Sustainable success comes from repeatedly finding and fixing your weaknesses. DORA's framework can help you determine which capabilities to focus on next for the biggest performance boost.

We hope the Accelerate State of DevOps Report helps organizations of all sizes, industries, and regions improve their DevOps capabilities, and we look forward to hearing your thoughts and feedback. To learn more about the report and implementing DevOps with Google Cloud:

", "url": "https://cloud.google.com/blog/products/devops-sre/announcing-the-2023-state-of-devops-report/", "title": "2023 State of DevOps Report: Culture is everything", "date_modified": "2023-10-05T11:00:03.000Z" }, { "content_html": "Comments", "url": "https://blog.bittacklr.be/the-workflow-pattern.html", "title": "The Workflow Pattern", "date_modified": "2023-10-05T08:17:21.000Z" }, { "content_html": "Comments", "url": "https://github.com/shuttle-hq/shuttle", "title": "Show HN: Shuttle – Build and ship backends without writing infrastructure files", "date_modified": "2023-10-04T11:51:17.000Z" }, { "content_html": "Comments", "url": "https://github.com/kata-containers/", "title": "Kata Containers: The speed of containers, the security of VMs", "date_modified": "2023-10-04T08:24:23.000Z" }, { "content_html": "Comments", "url": "https://xata.io/blog/pgroll-schema-migrations-postgres", "title": "Pgroll: zero-downtime, undoable, schema migrations for Postgres", "date_modified": "2023-10-03T14:20:56.000Z" }, { "content_html": "Comments", "url": "https://www.aidemos.info/dalle-3-examples/", "title": "Dalle-3 Examples", "date_modified": "2023-10-02T18:30:54.000Z" }, { "content_html": "Comments", "url": "https://floorp.app/en/", "title": "Floorp – a customisable Firefox fork from Japan", "date_modified": "2023-10-02T06:52:28.000Z" }, { "content_html": "Comments", "url": "https://github.com/NAlexPear/pg_branch", "title": "Pg_branch: Experimental Postgres extension brings Neon-like branching", "date_modified": "2023-10-02T03:49:35.000Z" }, { "content_html": "Comments", "url": "https://buildkite.com/blog/goodbye-integers-hello-uuids", "title": "Goodbye integers, hello UUIDv7", "date_modified": "2023-10-02T01:44:59.000Z" }, { "content_html": "Comments", "url": "https://abidmoon.hashnode.dev/2023-devops-is-terrible", "title": "2023 DevOps Is Terrible", "date_modified": "2023-10-01T18:29:54.000Z" }, { "content_html": "Comments", "url": "https://diziet.dreamwidth.org/16025.html", "title": "DKIM: Rotate and Publish Your Keys", "date_modified": "2023-10-01T07:44:29.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37717689", "title": "Ask HN: SaaS pricing pages with high prices and not “contact sales”", "date_modified": "2023-09-30T17:29:49.000Z" }, { "content_html": "Comments", "url": "https://github.com/ArchGPT/insomnium", "title": "Insomnium – Local, privacy-focused fork of Insomnia API client", "date_modified": "2023-09-29T18:49:16.000Z" }, { "content_html": "Comments", "url": "https://prdeving.wordpress.com/2023/09/29/mmo-architecture-source-of-truth-dataflows-i-o-bottlenecks-and-how-to-solve-them/", "title": "MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve", "date_modified": "2023-09-29T12:09:04.000Z" }, { "content_html": "Comments", "url": "https://posthog.com/blog/dev-marketing-paid-ads", "title": "Burning money on paid ads for a dev tool – what we've learned", "date_modified": "2023-09-29T08:30:03.000Z" }, { "content_html": "Comments", "url": "https://www.dockerstatus.com/", "title": "Docker Hub Registry timming out", "date_modified": "2023-09-28T20:12:31.000Z" }, { "content_html": "

Last year, we announced the Browser Rendering API – letting users running Puppeteer, a browser automation library, directly in Workers. Puppeteer is one of the most popular libraries used to interact with a headless browser instance to accomplish tasks like taking screenshots, generating PDFs, crawling web pages, and testing web applications. We’ve heard from developers that configuring and maintaining their own serverless browser automation systems can be quite painful.

The Workers Browser Rendering API solves this. It makes the Puppeteer library available directly in your Worker, connected to a real web browser, without the need to configure and manage infrastructure or keep browser sessions warm yourself. You can use @cloudflare/puppeteer to run the full Puppeteer API directly on Workers!

We’ve seen so much interest from the developer community since launching last year. While the Browser Rendering API is still in beta (sign up to our waitlist to get access), we wanted to share a way to get more out of our current limits by using the Browser Rendering API with Durable Objects. We’ll also be sharing pricing for the Rendering API, so you can build knowing exactly what you’ll pay for.

Building a responsive web design testing tool with the Browser Rendering API

As a designer or frontend developer, you want to make sure that content is well-designed for visitors browsing on different screen sizes. With the number of possible devices that users are browsing on are growing, it becomes difficult to test all the possibilities manually. While there are many testing tools on the market, we want to show how easy it is to create your own Chromium based tool with the Workers Browser Rendering API and Durable Objects.

We’ll be using the Worker to handle any incoming requests, pass them to the Durable Object to take screenshots and store them in an R2 bucket. The Durable Object is used to create a browser session that’s persistent. By using Durable Object Alarms we can keep browsers open for longer and reuse browser sessions across requests.

Let’s dive into how we can build this application:

  1. Create a Worker with a Durable Object, Browser Rendering API binding and R2 bucket. This is the resulting wrangler.toml:
name = \"rendering-api-demo\"\nmain = \"src/index.js\"\ncompatibility_date = \"2023-09-04\"\ncompatibility_flags = [ \"nodejs_compat\"]\naccount_id = \"c05e6a39aa4ccdd53ad17032f8a4dc10\"\n\n\n# Browser Rendering API binding\nbrowser = { binding = \"MYBROWSER\" }\n\n# Bind an R2 Bucket\n[[r2_buckets]]\nbinding = \"BUCKET\"\nbucket_name = \"screenshots\"\n\n# Binding to a Durable Object\n[[durable_objects.bindings]]\nname = \"BROWSER\"\nclass_name = \"Browser\"\n\n[[migrations]]\ntag = \"v1\" # Should be unique for each entry\nnew_classes = [\"Browser\"] # Array of new classes\n

2. Define the Worker

This Worker simply passes the request onto the Durable Object.

export default {\n\tasync fetch(request, env) {\n\n\t\tlet id = env.BROWSER.idFromName(\"browser\");\n\t\tlet obj = env.BROWSER.get(id);\n\t  \n\t\t// Send a request to the Durable Object, then await its response.\n\t\tlet resp = await obj.fetch(request.url);\n\t\tlet count = await resp.text();\n\t  \n\t\treturn new Response(\"success\");\n\t}\n};\n

3. Define the Durable Object class

const KEEP_BROWSER_ALIVE_IN_SECONDS = 60;\n\nexport class Browser {\n\tconstructor(state, env) {\n\t\tthis.state = state;\n\t\tthis.env = env;\n\t\tthis.keptAliveInSeconds = 0;\n\t\tthis.storage = this.state.storage;\n\t}\n  \n\tasync fetch(request) {\n\t\t// screen resolutions to test out\n\t\tconst width = [1920, 1366, 1536, 360, 414]\n\t\tconst height = [1080, 768, 864, 640, 896]\n\n\t\t// use the current date and time to create a folder structure for R2\n\t\tconst nowDate = new Date()\n\t\tvar coeff = 1000 * 60 * 5\n\t\tvar roundedDate = (new Date(Math.round(nowDate.getTime() / coeff) * coeff)).toString();\n\t\tvar folder = roundedDate.split(\" GMT\")[0]\n\n\t\t//if there's a browser session open, re-use it\n\t\tif (!this.browser) {\n\t\t\tconsole.log(`Browser DO: Starting new instance`);\n\t\t\ttry {\n\t\t\t  this.browser = await puppeteer.launch(this.env.MYBROWSER);\n\t\t\t} catch (e) {\n\t\t\t  console.log(`Browser DO: Could not start browser instance. Error: ${e}`);\n\t\t\t}\n\t\t  }\n\t\t\n\t\t// Reset keptAlive after each call to the DO\n\t\tthis.keptAliveInSeconds = 0;\n\t\t\n\t\tconst page = await this.browser.newPage();\n\n\t\t// take screenshots of each screen size \n\t\tfor (let i = 0; i < width.length; i++) {\n\t\t\tawait page.setViewport({ width: width[i], height: height[i] });\n\t\t\tawait page.goto(\"https://workers.cloudflare.com/\");\n\t\t\tconst fileName = \"screenshot_\" + width[i] + \"x\" + height[i]\n\t\t\tconst sc = await page.screenshot({\n\t\t\t\tpath: fileName + \".jpg\"\n\t\t\t}\n\t\t\t);\n\n\t\t\tthis.env.BUCKET.put(folder + \"/\"+ fileName + \".jpg\", sc);\n\t\t  }\n\t\t\n\t\t// Reset keptAlive after performing tasks to the DO.\n\t\tthis.keptAliveInSeconds = 0;\n\n\t\t// set the first alarm to keep DO alive\n\t\tlet currentAlarm = await this.storage.getAlarm();\n\t\tif (currentAlarm == null) {\n\t\tconsole.log(`Browser DO: setting alarm`);\n\t\tconst TEN_SECONDS = 10 * 1000;\n\t\tthis.storage.setAlarm(Date.now() + TEN_SECONDS);\n\t\t}\n\t\t\n\t\tawait this.browser.close();\n\t\treturn new Response(\"success\");\n\t}\n\n\tasync alarm() {\n\t\tthis.keptAliveInSeconds += 10;\n\t\n\t\t// Extend browser DO life\n\t\tif (this.keptAliveInSeconds < KEEP_BROWSER_ALIVE_IN_SECONDS) {\n\t\t  console.log(`Browser DO: has been kept alive for ${this.keptAliveInSeconds} seconds. Extending lifespan.`);\n\t\t  this.storage.setAlarm(Date.now() + 10 * 1000);\n\t\t} else console.log(`Browser DO: cxceeded life of ${KEEP_BROWSER_ALIVE_IN_SECONDS}. Browser DO will be shut down in 10 seconds.`);\n\t  }\n\n  }\n

That’s it! With less than a hundred lines of code, you can fully customize a powerful tool to automate responsive web design testing. You can even incorporate it into your CI pipeline to automatically test different window sizes with each build and verify the result is as expected by using an automated library like pixelmatch.

How much will this cost?

We’ve spoken to many customers deploying a Puppeteer service on their own infrastructure, on public cloud containers or functions or using managed services. The common theme that we’ve heard is that these services are costly – costly to maintain and expensive to run.

While you won’t be billed for the Browser Rendering API yet, we want to be transparent with you about costs you start building. We know it’s important to understand the pricing structure so that you don’t get a surprise bill and so that you can design your application efficiently.

You pay based on two usage metrics:

  1. Number of sessions: A Browser Session is a new instance of a browser being launched
  2. Number of concurrent sessions: Concurrent Sessions is the number of browser instances open at once

Using Durable Objects to persist browser sessions improves performance by eliminating the time that it takes to spin up a new browser session. Since it re-uses sessions, it cuts down on the number of concurrent sessions needed. We highly encourage this model of session re-use if you expect to see consistent traffic for applications that you build on the Browser Rendering API.

If you have feedback about this pricing, we’re all ears. Feel free to reach out through Discord (channel name: browser-rendering-api-beta) and share your thoughts.

Get Started

Sign up to our waitlist to get access to the Workers Browser Rendering API. We’re so excited to see what you build! Share your creations with us on Twitter/X @CloudflareDev or on our Discord community.

", "url": "https://blog.cloudflare.com/running-serverless-puppeteer-workers-durable-objects", "title": "Running Serverless Puppeteer with Workers and Durable Objects", "date_modified": "2023-09-28T13:00:38.000Z" }, { "content_html": "Comments", "url": "https://macoscontainers.org/", "title": "macOS Containers v0.0.1", "date_modified": "2023-09-26T07:01:45.000Z" }, { "content_html": "Comments", "url": "https://major.io/p/quadlets-replace-docker-compose/", "title": "Quadlets might make me finally stop using Docker-compose – Major Hayden", "date_modified": "2023-09-26T05:39:19.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37649222", "title": "Ask HN: Is “Distributed CI” Possible?", "date_modified": "2023-09-25T19:25:28.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37639010", "title": "Tell HN: There is a highlights page on HN", "date_modified": "2023-09-25T02:17:39.000Z" }, { "content_html": "Comments", "url": "https://adriano.fyi/posts/2023-09-24-choose-postgres-queue-technology/", "title": "Choose Postgres queue technology", "date_modified": "2023-09-24T20:30:01.000Z" }, { "content_html": "Comments", "url": "https://blog.liblab.com/typescript-npm-packages-done-right/", "title": "TypeScript NPM Packages Done Right", "date_modified": "2023-09-24T06:28:30.000Z" }, { "content_html": "

Comments

", "url": "https://semgrep.dev/blog/2021/semgrep-a-static-analysis-journey/", "title": "Semgrep: a static analysis journey", "date_modified": "2023-09-23T23:49:50.000Z" }, { "content_html": "Comments", "url": "https://bottlerocket.dev", "title": "Bottlerocket – Minimal, immutable Linux OS with verified boot", "date_modified": "2023-09-23T19:44:56.000Z" }, { "content_html": "Comments", "url": "https://blog.yossarian.net/2023/09/22/GitHub-Actions-could-be-so-much-better", "title": "GitHub Actions could be so much better", "date_modified": "2023-09-22T14:21:22.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/kb/1236/kubernetes-operator/", "title": "Tailscale Kubernetes Operator", "date_modified": "2023-09-22T12:39:06.000Z" }, { "content_html": "Comments", "url": "https://docs.getunleash.io/topics/feature-flags/feature-flag-best-practices", "title": "11 Principles for building and scaling feature flag systems", "date_modified": "2023-09-22T12:31:12.000Z" }, { "content_html": "Comments", "url": "https://www.sequoiacap.com/article/generative-ai-act-two/", "title": "Generative AI's Act Two", "date_modified": "2023-09-20T19:02:24.000Z" }, { "content_html": "Comments", "url": "https://strada.hotwired.dev/", "title": "Strada – Create fully native controls, driven by your web app", "date_modified": "2023-09-20T18:00:26.000Z" }, { "content_html": "Comments", "url": "https://github.com/opentofu", "title": "OpenTF renames itself to OpenTofu", "date_modified": "2023-09-20T06:44:42.000Z" }, { "content_html": "

Comments

", "url": "https://secluded.site/lxd-containers-for-human-beings/", "title": "LXD: Containers for Human Beings", "date_modified": "2023-09-19T22:01:36.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37570929", "title": "Show HN: Graphite – Stacked Diffs on GitHub", "date_modified": "2023-09-19T15:03:24.000Z" }, { "content_html": "

Comments

", "url": "https://blog.cyplo.dev/posts/2023/09/act-runner-image/", "title": "Creating a base OCI image for Nix flake builds within Gitea/Forgejo", "date_modified": "2023-09-18T16:13:54.000Z" }, { "content_html": "Comments", "url": "https://blog.the-pans.com/notes-on-the-foundationdb-paper/", "title": "How FoundationDB works and why it works (2021)", "date_modified": "2023-09-18T04:00:56.000Z" }, { "content_html": "Comments", "url": "https://www.subdomain.center/", "title": "Subdomain.center – discover all subdomains for a domain", "date_modified": "2023-09-16T03:32:27.000Z" }, { "content_html": "Comments", "url": "https://www.helloinbox.net", "title": "Show HN: Hello Inbox – Free email deliverability checklist for marketers", "date_modified": "2023-09-16T01:18:52.000Z" }, { "content_html": "Comments", "url": "https://earthly.dev/blog/shutting-down-earthly-ci/", "title": "We built the fastest CI and it failed", "date_modified": "2023-09-12T14:07:08.000Z" }, { "content_html": "Comments", "url": "https://msprout.notion.site/LogoScale-A-Method-for-Vectorizing-Small-Crappy-Logos-dc0035b7473c44f8b94ffc4026d286c0", "title": "LogoScale – A method for vectorizing small, crappy logos", "date_modified": "2023-09-08T23:38:39.000Z" }, { "content_html": "Comments", "url": "https://bun.sh/blog/bun-v1.0", "title": "Bun v1.0.0", "date_modified": "2023-09-08T14:41:03.000Z" }, { "content_html": "Comments", "url": "https://rivet.ironcladapp.com/", "title": "Show HN: Rivet – open-source AI Agent dev env with real-world applications", "date_modified": "2023-09-08T13:29:05.000Z" }, { "content_html": "Comments", "url": "https://jpcamara.com/2023/04/12/pgbouncer-is-useful.html", "title": "PgBouncer is useful, important, and fraught with peril", "date_modified": "2023-09-08T09:43:22.000Z" }, { "content_html": "Comments", "url": "https://napi.rs/", "title": "Napi: Build compiled Node.js add-ons in Rust", "date_modified": "2023-09-08T07:27:12.000Z" }, { "content_html": "Comments", "url": "https://github.com/leandromoreira/linux-network-performance-parameters", "title": "Linux network performance parameters", "date_modified": "2023-09-06T11:56:35.000Z" }, { "content_html": "

Vercel implemented Cron Jobs using Amazon EventBridge Scheduler, enabling their customers to create, manage, and run scheduled tasks at scale. The adoption of this feature was rapid, reaching over 7 million weekly cron invocations within a few months of release. This article shows how they did it and how they handle the massive scale they’re experiencing.

\n

Vercel builds a front-end cloud that makes it easier for engineers to deploy and run their front-end applications. With more than 100 million deployments in Vercel in the last two years, Vercel helps users take advantage of best-in-class AWS infrastructure with zero configuration by relying heavily on serverless technology. Vercel provides a lot of features that help developers host their front-end applications. However, until the beginning of this year, they hadn’t built Cron Jobs yet.

\n

A cron job is a scheduled task that automates running specific commands or scripts at predetermined intervals or fixed times. It enables users to set up regular, repetitive actions, such as backups, sending notification emails to customers, or processing payments when a subscription needs to be renewed. Cron jobs are widely used in computing environments to improve efficiency and automate routine operations, and they were a commonly requested feature from Vercel’s customers.

\n

In December 2022, Vercel hosted an internal hackathon to foster innovation. That’s where Vincent Voyer and Andreas Schneider joined forces to build a prototype cron job feature for the Vercel platform. They formed a team of five people and worked on the feature for a week. The team worked on different tasks, from building a user interface to display the cron jobs to creating the backend implementation of the feature.

\n

Amazon EventBridge Scheduler
When the hackathon team started thinking about solving the cron job problem, their first idea was to use Amazon EventBridge rules that run on a schedule. However, they realized quickly that this feature has a limit of 300 rules per account per AWS Region, which wasn’t enough for their intended use. Luckily, one of the team members had read the announcement of Amazon EventBridge Scheduler in the AWS Compute blog and they thought this would be a perfect tool for their problem.

\n

By using EventBridge Scheduler, they could schedule one-time or recurrently millions of tasks across over 270 AWS services without provisioning or managing the underlying infrastructure.

\n

\"How

\n

For creating a new cron job in Vercel, a customer needs to define the frequency in which this task will run and the API they want to invoke. Vercel, in the backend, uses EventBridge Scheduler and creates a new schedule when a new cron job is created.

\n

To call the endpoint, the team used an AWS Lambda function that receives the path that needs to be invoked as input parameters.

\n

\"How

\n

When the time comes for the cron job to run, EventBridge Scheduler invokes the function, which then calls the customer website endpoint that was configured.

\n

By the end of the week, Vincent and his team had a working prototype version of the cron jobs feature, and they won a prize at the hackathon.

\n

Building Vercel Cron Jobs
After working for one week on this prototype in December, the hackathon ended, and Vincent and his team returned to their regular jobs. In early January 2023, Vincent and the Vercel team decided to take the project and turn it into a real product.

\n

During the hackathon, the team built the fundamental parts of the feature, but there were some details that they needed to polish to make it production ready. Vincent and Andreas worked on the feature, and in less than two months, on February 22, 2023, they announced Vercel Cron Jobs to the public. The announcement tweet got over 400 thousand views, and the community loved the launch.

\n

\"Tweet

\n

The adoption of this feature was very rapid. Within a few months of launching Cron Jobs, Vercel reached over 7 million cron invocations per week, and they expect the adoption to continue growing.

\n

\"Cron

\n

How Vercel Cron Jobs Handles Scale
With this pace of adoption, scaling this feature is crucial for Vercel. In order to scale the amount of cron invocations at this pace, they had to make some business and architectural decisions.

\n

From the business perspective, they defined limits for their free-tier customers. Free-tier customers can create a maximum of two cron jobs in their account, and they can only have hourly schedules. This means that free customers cannot run a cron job every 30 minutes; instead, they can do it at most every hour. Only customers on Vercel paid tiers can take advantage of EventBridge Scheduler minute granularity for scheduling tasks.

\n

Also, for free customers, minute precision isn’t guaranteed. To achieve this, Vincent took advantage of the time window configuration from EventBridge Scheduler. The flexible time window configuration allows you to start a schedule within a window of time. This means that the scheduled tasks are dispersed across the time window to reduce the impact of multiple requests on downstream services. This is very useful if, for example, many customers want to run their jobs at midnight. By using the flexible time window, the load can spread across a set window of time.

\n

From the architectural perspective, Vercel took advantage of hosting the APIs and owning the functions that the cron jobs invoke.

\n

\"Validating

\n

This means that when the Lambda function is started by EventBridge Scheduler, the function ends its run without waiting for a response from the API. Then Vercel validates if the cron job ran by checking if the API and Vercel function ran correctly from its observability mechanisms. In this way, the function duration is very short, less than 400 milliseconds. This allows Vercel to run a lot of functions per second without affecting their concurrency limits.

\n

\"Lambda

\n

What Was The Impact?
Vercel’s implementation of Cron Jobs is an excellent example of what serverless technologies enable. In two months, with two people working full time, they were able to launch a feature that their community needed and enthusiastically adopted. This feature shows the completeness of Vercel’s platform and is an important feature to convince their customers to move to a paid account.

\n

If you want to get started with EventBridge Scheduler, see Serverless Land patterns for EventBridge Scheduler, where you’ll find a broad range of examples to help you.

\n

— Marcia

", "url": "https://aws.amazon.com/blogs/aws/how-vercel-shipped-cron-jobs-in-2-months-using-amazon-eventbridge-scheduler/", "title": "How Vercel Shipped Cron Jobs in 2 Months Using Amazon EventBridge Scheduler", "date_modified": "2023-09-05T20:04:30.000Z" }, { "content_html": "Comments", "url": "https://maxibestof.one/typefaces", "title": "Browse Websites by Fonts", "date_modified": "2023-09-05T17:24:28.000Z" }, { "content_html": "Comments", "url": "https://vale.sh/", "title": "Vale.sh – A Linter for Prose", "date_modified": "2023-09-03T15:58:18.000Z" }, { "content_html": "Comments", "url": "https://brooker.co.za/blog/2023/07/28/ds-testing.html", "title": "Invariants: A Better Debugger?", "date_modified": "2023-09-02T04:21:25.000Z" }, { "content_html": "Comments", "url": "https://avestura.dev/blog/explaining-the-postgres-meme", "title": "Explaining the Postgres iceberg", "date_modified": "2023-09-01T22:43:55.000Z" }, { "content_html": "

Comments

", "url": "https://www.avestura.dev/blog/explaining-the-postgres-meme", "title": "Explaining The Postgres Meme", "date_modified": "2023-09-01T22:41:09.000Z" }, { "content_html": "Comments", "url": "https://www.shuttle.rs/blog/2022/07/28/patterns-with-rust-types", "title": "Patterns with Rust Types", "date_modified": "2023-09-01T13:16:17.000Z" }, { "content_html": "Comments", "url": "https://www.mux.com/blog/what-are-react-server-components", "title": "Things I wish I knew before moving 50K lines of code to React Server Components", "date_modified": "2023-09-01T01:29:15.000Z" }, { "content_html": "Comments", "url": "https://determinate.systems/posts/flake-schemas", "title": "Show HN: Flake schemas – teaching Nix about your flake outputs", "date_modified": "2023-08-31T15:35:48.000Z" }, { "content_html": "Comments", "url": "https://matthiasportzel.com/docker/", "title": "Docker Is Four Things", "date_modified": "2023-08-31T12:11:47.000Z" }, { "content_html": "Comments", "url": "https://github.com/opentffoundation/roadmap/issues/24", "title": "HashiCorp silently amend Terraform Registry TOS", "date_modified": "2023-08-31T09:19:45.000Z" }, { "content_html": "Comments", "url": "https://taras.glek.net/post/lightweight-virtualization-metallize-libkrun-vsock/", "title": "Lightweight Virtualization: Libkrun, Vsock, Metallize", "date_modified": "2023-08-30T16:09:40.000Z" }, { "content_html": "Comments", "url": "https://gist.github.com/kj800x/be3001c07c49fdb36970633b0bc6defb", "title": "Hacking the LG Monitor's EDID", "date_modified": "2023-08-30T15:24:08.000Z" }, { "content_html": "Comments", "url": "https://buildkite.com/blog/applying-sre-principles-to-cicd", "title": "Applying SRE Principles to CI/CD", "date_modified": "2023-08-30T02:24:31.000Z" }, { "content_html": "Comments", "url": "https://bit.kevinslin.com/p/opentelemetry-in-2023", "title": "OpenTelemetry in 2023", "date_modified": "2023-08-28T14:58:09.000Z" }, { "content_html": "Comments", "url": "https://seacrew.github.io/helm-compose/", "title": "Helm-Compose – The Docker-compose like tool for K8s development", "date_modified": "2023-08-28T13:48:44.000Z" }, { "content_html": "Comments", "url": "https://newsletter.systemdesign.one/p/whatsapp-engineering", "title": "Why WhatsApp Was Able to Support 50B Messages a Day with 32 Engineers", "date_modified": "2023-08-27T09:42:34.000Z" }, { "content_html": "Comments", "url": "https://opentf.org/announcement", "title": "OpenTF Announces Fork of Terraform", "date_modified": "2023-08-25T14:56:48.000Z" }, { "content_html": "Comments", "url": "https://www.usenix.org/publications/loginonline/freebsd-firecracker", "title": "FreeBSD on Firecracker", "date_modified": "2023-08-24T18:52:46.000Z" }, { "content_html": "

Comments

", "url": "https://www.usenix.org/publications/loginonline/freebsd-firecracker", "title": "FreeBSD on Firecracker", "date_modified": "2023-08-24T08:21:13.000Z" }, { "content_html": "Comments", "url": "https://generated.photos/human-generator/", "title": "AI Real-Time Human Full-Body Photo Generator", "date_modified": "2023-08-23T15:28:45.000Z" }, { "content_html": "Comments", "url": "https://pglocks.org/", "title": "PostgreSQL Lock Conflicts", "date_modified": "2023-08-22T21:53:07.000Z" }, { "content_html": "Comments", "url": "https://godly.website/", "title": "Godly – Astronomically good web design inspiration", "date_modified": "2023-08-22T18:33:43.000Z" }, { "content_html": "Comments", "url": "https://www.points411.com/", "title": "Show HN: Points and Miles Database", "date_modified": "2023-08-22T15:43:13.000Z" }, { "content_html": "Comments", "url": "https://flakehub.com/", "title": "Show HN: FlakeHub – Discover and publish Nix flakes", "date_modified": "2023-08-22T15:29:24.000Z" }, { "content_html": "Comments", "url": "https://frunc.de/how-to/chrome-to-firefox-tips/", "title": "Switching from Chrome to Firefox", "date_modified": "2023-08-22T12:20:14.000Z" }, { "content_html": "Comments", "url": "https://blog.podman.io/2023/08/celebrating-500k-downloads-the-podman-desktop-journey-%f0%9f%8e%89/", "title": "Podman Desktop celebrates 500k downloads", "date_modified": "2023-08-21T23:44:54.000Z" }, { "content_html": "Comments", "url": "https://nivenly.org/blog/2023/08/19/an-announcement-regarding-kris-n%C3%B3va/", "title": "Kris Nova passed away", "date_modified": "2023-08-20T14:30:31.000Z" }, { "content_html": "Comments", "url": "https://github.com/rivet-gg/rivet", "title": "Show HN: Rivet – Open-source game server management with Nomad and Rust", "date_modified": "2023-08-19T13:38:22.000Z" }, { "content_html": "Comments", "url": "https://github.com/systeminit/si", "title": "System Initiative: remove the papercuts from DevOps work", "date_modified": "2023-08-18T07:46:32.000Z" }, { "content_html": "Comments", "url": "https://www.koyeb.com/", "title": "Show HN: Run globally distributed full-stack apps on high-performance MicroVMs", "date_modified": "2023-08-17T10:45:45.000Z" }, { "content_html": "Comments", "url": "https://github.com/EuropeanRemote/european-remote-software-companies", "title": "Show HN: Repo with a list of 80 decent companies hiring remotely in Europe", "date_modified": "2023-08-16T10:09:00.000Z" }, { "content_html": "Comments", "url": "https://arstechnica.com/gadgets/2023/08/the-printers-that-require-ink-to-scan-and-fax/", "title": "Requiring ink to scan a document–yet another insult from the printer industry", "date_modified": "2023-08-15T20:06:17.000Z" }, { "content_html": "Comments", "url": "https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/", "title": "We reduced the cost of building Mastodon at Twitter-scale by 100x", "date_modified": "2023-08-15T17:54:13.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37134293", "title": "Show HN: Layerform – Open-source development environments using Terraform files", "date_modified": "2023-08-15T14:15:51.000Z" }, { "content_html": "Comments", "url": "https://github.com/devcontainers/templates", "title": "Microsoft Docker Development Container Templates", "date_modified": "2023-08-13T07:17:09.000Z" }, { "content_html": "Comments", "url": "https://old.reddit.com/r/startups/comments/15p8qrx/my_0100m0_in_5_years_story/", "title": "My $0->$100M->$0 in 5 years story", "date_modified": "2023-08-12T17:37:25.000Z" }, { "content_html": "Comments", "url": "https://graphite.dev/blog/how-an-aws-aurora-feature-cut-db-costs", "title": "Aurora I/O optimized config saved 90% DB cost", "date_modified": "2023-08-10T18:29:48.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=37062007", "title": "Show HN: Infracost (YC W21): Be proactive with your cloud costs", "date_modified": "2023-08-09T13:01:14.000Z" }, { "content_html": "Comments", "url": "https://www.theregister.com/2023/08/08/amazon_arm_servers/", "title": "Amazon has more than half of all Arm server CPUs in the world", "date_modified": "2023-08-09T07:05:20.000Z" }, { "content_html": "Comments", "url": "https://supabase.com/blog/supabase-local-dev", "title": "Supabase Local Dev: migrations, branching, and observability", "date_modified": "2023-08-09T06:40:39.000Z" }, { "content_html": "Comments", "url": "https://packaging-con.org", "title": "PackagingCon – a conference only for software package management", "date_modified": "2023-08-08T20:59:18.000Z" }, { "content_html": "Comments", "url": "https://packaging-con.org", "title": "PackagingCon – A conference only for software package management", "date_modified": "2023-08-08T20:59:18.000Z" }, { "content_html": "Comments", "url": "https://dukope.itch.io/lcd-please", "title": "LCD, Please – de-make of “Papers, please”, celebrating 10 years since launch", "date_modified": "2023-08-08T17:30:09.000Z" }, { "content_html": "Comments", "url": "https://www.ory.dev/global-identity-and-access-management-multi-region/", "title": "Show HN: Blueprint for a distributed multi-region IAM with Go and CockroachDB", "date_modified": "2023-08-08T13:05:53.000Z" }, { "content_html": "Comments", "url": "https://tinloof.com/blog/the-seo-scam-62000-dollars-later", "title": "The SEO scam: $62,000 later", "date_modified": "2023-08-08T06:58:50.000Z" }, { "content_html": "Comments", "url": "https://discuss.linuxcontainers.org/t/introducing-incus/17781", "title": "Incus, a community fork of LXD, now part of Linux Containers", "date_modified": "2023-08-07T17:05:22.000Z" }, { "content_html": "Comments", "url": "https://github.com/supabase/postgres_lsp", "title": "Postgres Language Server", "date_modified": "2023-08-06T10:26:54.000Z" }, { "content_html": "Comments", "url": "https://jamesconroyfinn.com/github-actions-and-vanity-metrics", "title": "GitHub Actions and Vanity Metrics", "date_modified": "2023-08-05T09:37:09.000Z" }, { "content_html": "Comments", "url": "https://hocus.dev/blog/virtualizing-development-environments/", "title": "Virtualizing Development Environments in 2023", "date_modified": "2023-08-04T15:28:04.000Z" }, { "content_html": "Comments", "url": "https://hocus.dev/blog/virtualizing-development-environments/", "title": "Virtualizing development environments in 2023", "date_modified": "2023-08-04T15:28:04.000Z" }, { "content_html": "Comments", "url": "https://hydra-so.notion.site/Hydra-1-0-beta-318504444825401e8ce21796dcadd589", "title": "Show HN: Hydra 1.0 – open-source column-oriented Postgres", "date_modified": "2023-08-03T16:19:07.000Z" }, { "content_html": "Comments", "url": "https://ai.meta.com/blog/audiocraft-musicgen-audiogen-encodec-generative-ai-audio/", "title": "Meta Open Sources AudioCraft: Generative AI for Audio", "date_modified": "2023-08-02T15:36:31.000Z" }, { "content_html": "
\"Hardening
\"Hardening

Over the last couple of months, Workers KV has suffered from a series of incidents, culminating in three back-to-back incidents during the week of July 17th, 2023. These incidents have directly impacted customers that rely on KV — and this isn’t good enough.

We’re going to share the work we have done to understand why KV has had such a spate of incidents and, more importantly, share in depth what we’re doing to dramatically improve how we deploy changes to KV going forward.

Workers KV?

Workers KV — or just “KV” — is a key-value service for storing data: specifically, data with high read throughput requirements. It’s especially useful for user configuration, service routing, small assets and/or authentication data.

We use KV extensively inside Cloudflare too, with Cloudflare Access (part of our Zero Trust suite) and Cloudflare Pages being some of our highest profile internal customers. Both teams benefit from KV’s ability to keep regularly accessed key-value pairs close to where they’re accessed, as well its ability to scale out horizontally without any need to become an expert in operating KV.

Given Cloudflare’s extensive use of KV, it wasn’t just external customers impacted. Our own internal teams felt the pain of these incidents, too.

The summary of the post-mortem

Back in June 2023, we announced the move to a new architecture for KV, which is designed to address two major points of customer feedback we’ve had around KV: high latency for infrequently accessed keys (or a key accessed in different regions), and working to ensure the upper bound on KV’s eventual consistency model for writes is 60 seconds — not “mostly 60 seconds”.

At the time of the blog, we’d already been testing this internally, including early access with our community champions and running a small % of production traffic to validate stability and performance expectations beyond what we could emulate within a staging environment.

However, in the weeks between mid-June and culminating in the series of incidents during the week of July 17th, we would continue to increase the volume of new traffic onto the new architecture. When we did this, we would encounter previously unseen problems (many of these customer-impacting) — then immediately roll back, fix bugs, and repeat. Internally, we’d begun to identify that this pattern was becoming unsustainable — each attempt to cut traffic onto the new architecture would surface errors or behaviors we hadn’t seen before and couldn’t immediately explain, and thus we would roll back and assess.

The issues at the root of this series of incidents proved to be significantly challenging to track and observe. Once identified, the two causes themselves proved to be quick to fix, but an (1) observability gap in our error reporting and (2) a mutation to local state that resulted in an unexpected mutation of global state were both hard to observe and reproduce over the days following the customer-facing impact ending.

The detail

One important piece of context to understand before we go into detail on the post-mortem: Workers KV is composed of two separate Workers scripts – internally referred to as the Storage Gateway Worker and SuperCache. SuperCache is an optional path in the Storage Gateway Worker workflow, and is the basis for KV's new (faster) backend (refer to the blog).

Here is a timeline of events:

\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
TimeDescription
2023-07-17 21:52 UTCCloudflare observes alerts showing 500 HTTP status codes in the MEL01 data-center (Melbourne, AU) and begins investigating.
We also begin to see a small set of customers reporting HTTP 500s being returned via multiple channels. It is not immediately clear if this is a data-center-wide issue or KV specific, as there had not been a recent KV deployment, and the issue directly correlated with three data-centers being brought back online.
2023-07-18 00:09 UTCWe disable the new backend for KV in MEL01 in an attempt to mitigate the issue (noting that there had not been a recent deployment or change to the % of users on the new backend).
2023-07-18 05:42 UTCInvestigating alerts showing 500 HTTP status codes in VIE02 (Vienna, AT) and JNB01 (Johannesburg, SA).
2023-07-18 13:51 UTCThe new backend is disabled globally after seeing issues in VIE02 (Vienna, AT) and JNB01 (Johannesburg, SA) data-centers, similar to MEL01. In both cases, they had also recently come back online after maintenance, but it remained unclear as to why KV was failing.
2023-07-20 19:12 UTCThe new backend is inadvertently re-enabled while deploying the update due to a misconfiguration in a deployment script.
2023-07-20 19:33 UTCThe new backend is (re-) disabled globally as HTTP 500 errors return.
2023-07-20 23:46 UTCBroken Workers script pipeline deployed as part of gradual rollout due to incorrectly defined pipeline configuration in the deployment script.
Metrics begin to report that a subset of traffic is being black-holed.
2023-07-20 23:56 UTCBroken pipeline rolled back; errors rates return to pre-incident (normal) levels.

All timestamps referenced are in Coordinated Universal Time (UTC).

We initially observed alerts showing 500 HTTP status codes in the MEL01 data-center (Melbourne, AU) at 21:52 UTC on July 17th, and began investigating. We also received reports from a small set of customers reporting HTTP 500s being returned via multiple channels. This correlated with three data centers being brought back online, and it was not immediately clear if it related to the data centers or was KV-specific — especially given there had not been a recent KV deployment. On 05:42, we began investigating alerts showing 500 HTTP status codes in VIE02 (Vienna) and JNB02 (Johannesburg) data-centers; while both had recently come back online after maintenance, it was still unclear why KV was failing. At 13:51 UTC, we made the decision to disable the new backend globally.

Following the incident on July 18th, we attempted to deploy an allow-list configuration to reduce the scope of impacted accounts. However, while attempting to roll out a change for the Storage Gateway Worker at 19:12 UTC on July 20th, an older configuration was progressed causing the new backend to be enabled again, leading to the third event. As the team worked to fix this and deploy this configuration, they attempted to manually progress the deployment at 23:46 UTC, which resulted in the passing of a malformed configuration value that caused traffic to be sent to an invalid Workers script configuration.

After all deployments and the broken Workers configuration (pipeline) had been rolled back at 23:56 on the 20th July, we spent the following three days working to identify the root cause of the issue. We lacked observability as KV's Worker script (responsible for much of KV's logic) was throwing an unhandled exception very early on in the request handling process. This was further exacerbated by prior work to disable error reporting in a disabled data-center due to the noise generated, which had previously resulted in logs being rate-limited upstream from our service.

This previous mitigation prevented us from capturing meaningful logs from the Worker, including identifying the exception itself, as an uncaught exception terminates request processing. This has raised the priority of improving how unhandled exceptions are reported and surfaced in a Worker (see Recommendations, below, for further details). This issue was exacerbated by the fact that KV's Worker script would fail to re-enter its \"healthy\" state when a Cloudflare data center was brought back online, as the Worker was mutating an environment variable perceived to be in request scope, but that was in global scope and persisted across requests. This effectively left the Worker “frozen” with the previous, invalid configuration for the affected locations.

Further, the introduction of a new progressive release process for Workers KV, designed to de-risk rollouts (as an action from a prior incident), prolonged the incident. We found a bug in the deployment logic that led to a broader outage due to an incorrectly defined configuration.

This configuration effectively caused us to drop a single-digit % of traffic until it was rolled back 10 minutes later. This code is untested at scale, and we need to spend more time hardening it before using it as the default path in production.

Additionally: although the root cause of the incidents was limited to three Cloudflare data-centers (Melbourne, Vienna, and Johannesburg), traffic across these regions still uses these data centers to route reads and writes to our system of record. Because these three data centers participate in KV’s new backend as regional tiers, a portion of traffic across the Oceania, Europe, and African regions was affected. Only a portion of keys from enrolled namespaces use any given data center as a regional tier in order to limit a single (regional) point of failure, so while traffic across all data centers in the region was impacted, nowhere was all traffic in a given data center affected.

We estimated the affected traffic to be 0.2-0.5% of KV's global traffic (based on our error reporting), however we observed some customers with error rates approaching 20% of their total KV operations. The impact was spread across KV namespaces and keys for customers within the scope of this incident.

Both KV’s high total traffic volume and its role as a critical dependency for many customers amplify the impact of even small error rates. In all cases, once the changes were rolled back, errors returned to normal levels and did not persist.

Thinking about risks in building software

Before we dive into what we’re doing to significantly improve how we build, test, deploy and observe Workers KV going forward, we think there are lessons from the real world that can equally apply to how we improve the safety factor of the software we ship.

In traditional engineering and construction, there is an extremely common procedure known as a   “JSEA”, or Job Safety and Environmental Analysis (sometimes just “JSA”). A JSEA is designed to help you iterate through a list of tasks, the potential hazards, and most importantly, the controls that will be applied to prevent those hazards from damaging equipment, injuring people, or worse.

One of the most critical concepts is the “hierarchy of controls” — that is, what controls should be applied to mitigate these hazards. In most practices, these are elimination, substitution, engineering, administration and personal protective equipment. Elimination and substitution are fairly self-explanatory: is there a different way to achieve this goal? Can we eliminate that task completely? Engineering and administration ask us whether there is additional engineering work, such as changing the placement of a panel, or using a horizontal boring machine to lay an underground pipe vs. opening up a trench that people can fall into.

The last and lowest on the hierarchy, is personal protective equipment (PPE). A hard hat can protect you from severe injury from something falling from above, but it’s a last resort, and it certainly isn’t guaranteed. In engineering practice, any hazard that only lists PPE as a mitigating factor is unsatisfactory: there must be additional controls in place. For example, instead of only wearing a hard hat, we should engineer the floor of scaffolding so that large objects (such as a wrench) cannot fall through in the first place. Further, if we require that all tools are attached to the wearer, then it significantly reduces the chance the tool can be dropped in the first place. These controls ensure that there are multiple degrees of mitigation — defense in depth — before your hard hat has to come into play.

Coming back to software, we can draw parallels between these controls: engineering can be likened to improving automation, gradual rollouts, and detailed metrics. Similarly, personal protective equipment can be likened to code review: useful, but code review cannot be the only thing protecting you from shipping bugs or untested code. Automation with linters, more robust testing, and new metrics are all vastly safer ways of shipping software.

As we spent time assessing where to improve our existing controls and how to put new controls in place to mitigate risks and improve the reliability (safety) of Workers KV, we took a similar approach: eliminating unnecessary changes, engineering more resilience into our codebase, automation, deployment tooling, and only then looking at human processes.

How we plan to get better

Cloudflare is undertaking a larger, more structured review of KV's observability tooling, release infrastructure and processes to mitigate not only the contributing factors to the incidents within this report, but recent incidents related to KV. Critically, we see tooling and automation as the most powerful mechanisms for preventing incidents, with process improvements designed to provide an additional layer of protection. Process improvements alone cannot be the only mitigation.

Specifically, we have identified and prioritized the below efforts as the most important next steps towards meeting our own availability SLOs, and (above all) make KV a service that customers building on Workers can rely on for storing configuration and service data in the hot path of their traffic:

This is not an exhaustive list: we're continuing to expand on preventative measures associated with these and other incidents. These changes will not only improve KVs reliability, but other services across Cloudflare that KV relies on, or that rely on KV.

We recognize that KV hasn’t lived up to our customers’ expectations recently. Because we rely on KV so heavily internally, we’ve felt that pain first hand as well. The work to fix the issues that led to this cycle of incidents is already underway. That work will not only improve KV’s reliability but also improve the reliability of any software written on the Cloudflare Workers developer platform, whether by our customers or by ourselves.

", "url": "http://blog.cloudflare.com/workers-kv-restoring-reliability/", "title": "Hardening Workers KV", "date_modified": "2023-08-02T13:05:42.000Z" }, { "content_html": "Comments", "url": "https://zknill.io/posts/edge-database/", "title": "So, you want to deploy on the edge?", "date_modified": "2023-07-31T11:54:00.000Z" }, { "content_html": "

Comments

", "url": "https://matduggan.com/serverless-functions-post-mortem/", "title": "Serverless Functions Post-Mortem", "date_modified": "2023-07-28T20:37:02.000Z" }, { "content_html": "

We are introducing a new charge for public IPv4 addresses. Effective February 1, 2024 there will be a charge of $0.005 per IP per hour for all public IPv4 addresses, whether attached to a service or not (there is already a charge for public IPv4 addresses you allocate in your account but don’t attach to an EC2 instance).

\n

Public IPv4 Charge
As you may know, IPv4 addresses are an increasingly scarce resource and the cost to acquire a single public IPv4 address has risen more than 300% over the past 5 years. This change reflects our own costs and is also intended to encourage you to be a bit more frugal with your use of public IPv4 addresses and to think about accelerating your adoption of IPv6 as a modernization and conservation measure.

\n

This change applies to all AWS services including Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (RDS) database instances, Amazon Elastic Kubernetes Service (EKS) nodes, and other AWS services that can have a public IPv4 address allocated and attached, in all AWS regions (commercial, AWS China, and GovCloud). Here’s a summary in tabular form:

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Public IP Address TypeCurrent Price/Hour (USD)New Price/Hour (USD)
(Effective February 1, 2024)
In-use Public IPv4 address (including Amazon provided public IPv4 and Elastic IP) assigned to resources in your VPC, Amazon Global Accelerator, and AWS Site-to-site VPN tunnelNo charge$0.005
Additional (secondary) Elastic IP Address on a running EC2 instance$0.005$0.005
Idle Elastic IP Address in account$0.005$0.005
\n

The AWS Free Tier for EC2 will include 750 hours of public IPv4 address usage per month for the first 12 months, effective February 1, 2024. You will not be charged for IP addresses that you own and bring to AWS using Amazon BYOIP.

\n

Starting today, your AWS Cost and Usage Reports automatically include public IPv4 address usage. When this price change goes in to effect next year you will also be able to use AWS Cost Explorer to see and better understand your usage.

\n

As I noted earlier in this post, I would like to encourage you to consider accelerating your adoption of IPv6. A new blog post shows you how to use Elastic Load Balancers and NAT Gateways for ingress and egress traffic, while avoiding the use of a public IPv4 address for each instance that you launch. Here are some resources to show you how you can use IPv6 with widely used services such as EC2, Amazon Virtual Private Cloud (Amazon VPC), Amazon Elastic Kubernetes Service (EKS), Elastic Load Balancing, and Amazon Relational Database Service (RDS):

\n \n

Earlier this year we enhanced EC2 Instance Connect and gave it the ability to connect to your instances using private IPv4 addresses. As a result, you no longer need to use public IPv4 addresses for administrative purposes (generally using SSH or RDP).

\n

Public IP Insights
In order to make it easier for you to monitor, analyze, and audit your use of public IPv4 addresses, today we are launching Public IP Insights, a new feature of Amazon VPC IP Address Manager that is available to you at no cost. In addition to helping you to make efficient use of public IPv4 addresses, Public IP Insights will give you a better understanding of your security profile. You can see the breakdown of public IP types and EIP usage, with multiple filtering options:

\n

\n

You can also see, sort, filter, and learn more about each of the public IPv4 addresses that you are using:

\n

\n

Using IPv4 Addresses Efficiently
By using the new IP Insights tool and following the guidance that I shared above, you should be ready to update your application to minimize the effect of the new charge. You may also want to consider using AWS Direct Connect to set up a dedicated network connection to AWS.

\n

Finally, be sure to read our new blog post, Identify and Optimize Public IPv4 Address Usage on AWS, for more information on how to make the best use of public IPv4 addresses.

\n

— Jeff;

", "url": "https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/", "title": "New – AWS Public IPv4 Address Charge + Public IP Insights", "date_modified": "2023-07-28T18:01:23.000Z" }, { "content_html": "Comments", "url": "https://envoy-playground.apoxy.dev", "title": "Show HN: Envoy playground in the browser", "date_modified": "2023-07-27T19:59:45.000Z" }, { "content_html": "Comments", "url": "https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html", "title": "Building and operating a pretty big storage system called S3", "date_modified": "2023-07-27T15:20:35.000Z" }, { "content_html": "Comments", "url": "https://jcs.org/2023/07/12/api", "title": "Advice for Operating a Public-Facing API", "date_modified": "2023-07-22T00:36:29.000Z" }, { "content_html": "

Workforce Identity Federation allows use of an external identity provider (IdP) to authenticate and authorize users (including employees, partners, and contractors) to Google Cloud resources without provisioning identities in Cloud Identity. Before its introduction, only identities existing within Cloud Identity could be used with Cloud Identity Access Management (IAM). 

Here’s how to configure an example Javascript web application hosted in Google Cloud to call Google Cloud APIs after being authenticated with an Azure AD using Workforce Identity Federation.

Workforce Identity can be used with IdPs supporting OpenID Connect (OIDC) or SAML 2.0. You can read more about it in our blog post and product documentation page. 

Configuring Workforce Identity Federation

There will be three high level configuration steps required:

  1. Prepare your external IdP and get required configuration parameters.

  2. Create a logical container for your external identities in Google Cloud in the form of Workforce Identity Pool.

  3. Establish relation between your Workforce Identity Pool and external IdP by configuring WorkforceIdentity Pool Provider using information gathered in the first step.

Before Workforce Identity Federation can be used, a one-way trust relationship must be established between your Google Cloud environment and external IdP. This is achieved by configuring the following resources in Google Cloud: 1) a Workforce Identity Pool, which is a logical container for external identities and 2) a corresponding Workforce Identity Pool Provider, encapsulating technical details of external IdP integration.

Understanding of OIDC flow may be helpful to understand integration with your application code. We will focus on a single page web application which calls Google Cloud APIs. For simplicity, we omit details of protocol, such as audiences and claims, as they are irrelevant to understanding the flow:

\"1

  1. Client downloads a web app with JS code. In our example, static content is exposed from the GCS storage bucket.

  2. The unauthenticated user is redirected to an external IdP login page for authentication.

  3. On successful login, the external IdP returns the authentication result including an ID Token. 

  4. The ID Token contains information about identity and can be exchanged into an “access token”. This is accomplished on the Google Cloud side with a service called Secure Token Service (STS) (API documentation).

  5. STS verifies the ID Token and if successful returns a Google Identity access token.

  6. Access token can be used as a bearer token in subsequent Google Cloud API calls. Please note: by default, access tokens are good for 1 hour (3,600 seconds). When the access token has expired, your token management code must get a new one.

As an example of incorporating this flow within your application we will use Azure Active Directory as an external IdP.

IdP configuration

Azure AD requires following steps to act as IdP for Workforce Identity Federation:

  1. Registering application 

  2. Assigning users (and groups) to the enterprise application

Azure AD performs identity management only for registered applications. This is why the first thing we need to do is to create a new application registration (go to Azure Active Directory and select App registrations). An important parameter to assign is the type of redirect URI, in our case we choose “Single-page application (SPA)” If our goal is to provide integration with Google Cloud Federated Console (console.cloud.google) we need to choose a type of “Web”. More information about configuration and the federated version of Cloud Console is provided in Configure Azure AD-based workforce identity federation.

\"2
Registering an application in Azure Active Directory

Next step is to choose “ID Tokens” from the list of tokens issued by authorization endpoint. For details see OpenID Connect (OIDC). 

From the information screen after you finish registration note “Application (client) ID” (this is “client id” needed for Identity Workforce Pool Provider configuration in Google Cloud), and click on “Endpoints” above.

\"3

From the endpoints window copy the “OpenID Connect metadata document” URL, navigate to it and look for the “issuer” field. Value of this field (in the form of https://login.microsoftonline.com/TENANT_ID/v2.0) will be required for the Workforce Identity Federation).

\"4

Last step is to go to “Enterprise applications”, find your application and assign users and groups that you want to give access to your application.

Configure Google Cloud Workforce Identity Federation

Configurations steps to be executed on Google Cloud:

  1. Specify billing project

  2. Enable APIs

  3. Create Workforce Identity Pool

  4. Create Workforce Identity Pool Provider

  5. Assign required permissions to external identities from the Pool

Before you begin make sure APIs are enabled on the billing project (as Workforce Identity Federation artifacts are at organization level, you need to specify the project which will be used for billing associated with those resources): 

You can find detailed information in the “Before you begin” section of product documentation.

Create Workforce Identity Pool

code_block
[StructValue([(u'code', u'gcloud iam workforce-pools create $WORKFORCE_POOL_ID \\\\\\r\\n --organization=$ORGANIZATION_ID \\\\\\r\\n --description=\"$WORKFORCE_POOL_DESCR\" \\\\\\r\\n --location=global \\\\\\r\\n --billing-project=\"$PROJECT_ID\"'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e0d449ccad0>)])]

Please consult details of configuration parameters in the corresponding section of Cloud SDK documentation.

Create Workforce Identity Pool Provider

code_block
[StructValue([(u'code', u'gcloud iam workforce-pools providers create-oidc $WORKFORCE_PROVIDER_ID \\\\\\r\\n --workforce-pool=$WORKFORCE_POOL_ID \\\\\\r\\n --display-name=\"$WORKFORCE_PROVIDER_ID\" \\\\\\r\\n --description=\"$WORKFORCE_PROVIDER_ID\" \\\\\\r\\n --issuer-uri=\"$ISSUER_URI\" \\\\\\r\\n --client-id=\"$CLIENT_ID\" \\\\ --attribute-mapping=\"google.subject=assertion.preferred_username,google.groups=assertion.groups\" \\\\\\r\\n --location=global \\\\\\r\\n --billing-project=\"$PROJECT_ID\"'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e0d449cc750>)])]

This is a crucial step in the configuration, as here we establish one-way trust to our Idp. We will need information from the Azure environment: issuer uri and client id, we should have them from the previous step. 

Please note the attribute mapping parameter where we decide which attribute (assertion) from IdP we will use for the required google.subject attribute. In our example we use preferred_username assertion which in case of Azure AD carries email of authenticated user. This determines the syntax we will use for referencing external identities in Google Cloud as described in Represent workforce pool users in IAM policies.

Now we just need to assign the correct set of roles to our external identities. In the following example we assign the serviceUserConsumer role, which is required to consume any Google Cloud API, so will also be necessary for your external identities.

code_block
[StructValue([(u'code', u'gcloud projects add-iam-policy-binding $PROJECT_ID \\\\\\r\\n --role=\"roles/serviceusage.serviceUsageConsumer\" \\\\\\r\\n--member=\"principal://iam.googleapis.com/locations/global/workforcePools/$WORKFORCE_POOL_ID/subject/$TEST_SUBJECT\"'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e0d4519bd10>)])]

$TEST_SUBJECT in our case is an email of one of Azure AD users we assigned to our enterprise application.

Now our Workforce Identity Federation should be ready to rock’n’roll!

Example Web Application

To use Azure AD ID Tokens in a Web Application, Microsoft recommends using the Microsoft Authentication Library for JavaScript (MSAL.js). Several wrappers exist for this library to be used in e.g. Node.js, React and Angular. 

To demonstrate using the Workforce Identity Federation, the following example is provided in Javascript using the Microsoft Authentication Library for JavaScript (MSAL.js) 2.0 for Browser-Based Single-Page Applications.

As a first step, the MSAL.js library must be loaded and initialized. For simplicity we are using the CDN version of the library, in most cases the NPM package should be used instead. The script is loaded in the HTML body with async and defer options to ensure that it does not interfere with page loading time.

code_block
[StructValue([(u'code', u'<html>\\r\\n<body>\\r\\n <script\\r\\n async\\r\\n defer\\r\\n type=\"text/javascript\"\\r\\n src=\"https://alcdn.msauth.net/browser/2.3.33/js/msal-browser.min.js\"\\r\\n integrity=\"sha384-sIyEdUUBfHyE7nRpvFHA2odaB/z+HNe9frVsvOKkncXSRPnyek6HeakhZhNdJLrj\"\\r\\n crossorigin=\"anonymous\"></script>\\r\\n</body>\\r\\n</html>'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e0d4519b910>)])]

MSAL.js can use a popup or redirect login. Without a backend to handle the redirect, the popup method is the recommended method. To show a popup without triggering the popup blocker in modern browsers, the popup needs to be triggered by a user action (e.g. a button click) and must happen within a short period of time after the button was clicked.

As a result an HTML button is added to trigger the login popup and inline Javascript with a setup function called after the MSAL.js library is loaded, which enables the login button and initializes a PublicClientApplication from the MSAL.js library. The PublicClientApplication initialization requires the clientId and authority as defined in the AD Single Page Setup above.

code_block
[StructValue([(u'code', u'<html>\\r\\n <script>\\r\\n function setup() {\\r\\n const configuration = {\\r\\n auth: { \\r\\n clientId: process.env.CLIENT_ID, \\r\\n authority: process.env.AD_AUTHORITY\\r\\n }\\r\\n };\\r\\n const PublicClientApplication = new PublicClientApplication(configuration);\\r\\n document.getElementById(\\'login\\').disabled = false; \\r\\n }\\r\\n </script>\\r\\n<body>\\r\\n <button id=\"login\" disabled>login</button>\\r\\n <script\\r\\n async\\r\\n defer\\r\\n type=\"text/javascript\"\\r\\n src=\"https://alcdn.msauth.net/browser/2.33.0/js/msal-browser.min.js\"\\r\\n integrity=\"sha384-sIyEdUUBfHyE7nRpvFHA2odaB/z+HNe9frVsvOKkncXSRPnyek6HeakhZhNdJLrj\"\\r\\n crossorigin=\"anonymous\"\\r\\n onload=\"setup()\"\\r\\n ></script>\\r\\n</body>\\r\\n</html>'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e0d4519b210>)])]

Now the login functionality is added to open the login popup and handle the login response. The response contains an access token and an ID token. The ID token will be sent to STS to be exchanged for a Google Identity access token. The access token returned after successful login is intended to be used with Azure. 

In our example, it is used to query the Graph API to retrieve the user information from AD. To access Google Cloud resources we need to exchange an ID token from Azure for another access token using Google Cloud STS, based on trust established by the Workforce Identity Federation setup. The Google Identity access token can now be used to access Google APIs, like e.g. the resourcemanager API to retrieve the list of projects visible to the current user (requires that the user has the IAM permission resourcemanager.projects.get)

Please note that in calling Google Cloud STS we need to provide a proper audience, which is the URI of our Workforce Identity Federation pool provider.

code_block
[StructValue([(u'code', u'<html>\\r\\n <script>\\r\\n var publicClientApplication;\\r\\n function setup() {\\r\\n const configuration = {\\r\\n auth: { \\r\\n clientId: process.env.CLIENT_ID, \\r\\n authority: process.env.AD_AUTHORITY\\r\\n }\\r\\n };\\r\\n publicClientApplication = new msal.PublicClientApplication(configuration);\\r\\n document.getElementById(\\'login\\').disabled = false;\\r\\n }\\r\\n async function login() {\\r\\n const loginResponse = await publicClientApplication.loginPopup({\\r\\n scopes: [\"openid\", \"profile\", \"email\"],\\r\\n });\\r\\n if (loginResponse.idToken) {\\r\\n // call STS for token exchange\\r\\n const stsUrl = \"https://sts.googleapis.com/v1/token\";\\r\\n const body = JSON.stringify({\\r\\n grantType: \"urn:ietf:params:oauth:grant-type:token-exchange\",\\r\\n audience: \"//iam.googleapis.com/locations/global/workforcePools/my-workforce-pool/providers/my-test-provider\",\\r\\n scope: \"https://www.googleapis.com/auth/cloud-platform\",\\r\\n requestedTokenType: \"urn:ietf:params:oauth:token-type:access_token\",\\r\\n subjectToken: loginResponse.idToken,\\r\\n subjectTokenType: \"urn:ietf:params:oauth:token-type:id_token\",\\r\\n options: JSON.stringify({userProject:\"ffeldhaus-permanent\"})\\r\\n });\\r\\n const stsTokenResponse = await fetch(stsUrl, {\\r\\n method: \"POST\",\\r\\n headers: {\\r\\n \"Content-Type\": \"application/json\"\\r\\n },\\r\\n body: body\\r\\n });\\r\\n const stsTokenResponseJson = await stsTokenResponse.json();\\r\\n let response;\\r\\n let userInfo;\\r\\n userInfoUrl = \"https://graph.microsoft.com/oidc/userinfo\";\\r\\n accessToken = loginResponse.accessToken;\\r\\n response = await fetch(\\r\\n userInfoUrl,\\r\\n {\\r\\n headers: {\\r\\n authorization: `Bearer ${accessToken}`,\\r\\n },\\r\\n }\\r\\n );\\r\\n userInfo = await response.json();\\r\\n document.getElementById(\"userInfo\").innerHTML = \"<pre>\" + JSON.stringify(userInfo, null, 2) + \"</pre>\";\\r\\n listProjectsUrl = \"https://cloudresourcemanager.googleapis.com/v1/projects\";\\r\\n accessToken = stsTokenResponseJson.access_token;\\r\\n response = await fetch(\\r\\n listProjectsUrl,\\r\\n {\\r\\n headers: {\\r\\n authorization: `Bearer ${accessToken}`,\\r\\n },\\r\\n }\\r\\n );\\r\\n listProjects = await response.json();\\r\\n document.getElementById(\"projectList\").innerHTML = \"<pre>\" + JSON.stringify(listProjects, null, 2) + \"</pre>\";\\r\\n };\\r\\n }\\r\\n </script>\\r\\n <body>\\r\\n <button id=\"login\" onclick=\"login()\" disabled>login</button>\\r\\n <div id=\"userInfo\"></div>\\r\\n <div id=\"projectList\"></div>\\r\\n <script\\r\\n async\\r\\n defer\\r\\n type=\"text/javascript\"\\r\\n src=\"https://alcdn.msauth.net/browser/2.33.0/js/msal-browser.min.js\"\\r\\n integrity=\"sha384-sIyEdUUBfHyE7nRpvFHA2odaB/z+HNe9frVsvOKkncXSRPnyek6HeakhZhNdJLrj\"\\r\\n crossorigin=\"anonymous\"\\r\\n onload=\"setup()\"\\r\\n ></script>\\r\\n </body>\\r\\n</html>'), (u'language', u''), (u'caption', <wagtail.wagtailcore.rich_text.RichText object at 0x3e0d4519b350>)])]

The access token can then be used to authenticate against all APIs and services which support Workforce Identity Federation.

\"5

The code shared above must be hosted on a valid HTTPS endpoint which is configured in AD as an endpoint for the single page application. This can be achieved with GCLB and a public GCS bucket.

Users and Groups from Active Directory can be represented in IAM policies using the following format:

\"6

Important: All principals must have the role roles/serviceusage.serviceUsageConsumer which contains IAM permission serviceusage.services.use.

Troubleshooting

As with every software development exercise, sooner or later something goes wrong, here is the list of hints we think may be useful when things are not going as planned.

Familiarize yourself with IAM logging: Read the Example logs for Workforce Identity Federation section of IAM documentation.

Use Cloud Logging: This should always be the first thing to do, check the logs in Logs Explorer looking for errors, unauthorized calls, and so on. Remember, to view logs you need corresponding permissions on the project.

Check permissions: In the text we mentioned permissions required for the external identity to call Google Cloud APIs, check if your external identity has been assigned permissions including roles/serviceusage.serviceUsageConsumer role and roles associated with APIs being called. Check preliminary requirements again.

Check your audience: When calling STS for token exchange you must have the correct audience set, which is referring to your Workforce Identity Pool Provider, check our code example for syntax.

Related Article

Introducing Workforce Identity Federation to easily manage workforce access to Google Cloud

Workforce Identity Federation can help users onboard to Google Cloud using their identity and credentials that currently exist with their...

Read Article
", "url": "https://cloud.google.com/blog/products/identity-security/using-workforce-identity-federation-with-api-based-web-applications/", "title": "Using Workforce Identity Federation with API-based web applications", "date_modified": "2023-07-21T16:00:00.000Z" }, { "content_html": "Comments", "url": "https://webcache.googleusercontent.com/search?q=cache:2T9NpG8thZgJ:https://community.fly.io/t/service-interruption-cant-destroy-machine-deploy-or-restart/14227&cd=9&hl=en&ct=clnk&gl=au", "title": "Fly.io Postgres cluster down for 3 days, no word from them about it", "date_modified": "2023-07-20T23:42:10.000Z" }, { "content_html": "Comments", "url": "https://www.infoq.com/news/2023/07/linkedin-protocol-buffers-restli/", "title": "LinkedIn adopts protocol buffers and reduces latency up to 60%", "date_modified": "2023-07-19T22:33:52.000Z" }, { "content_html": "Comments", "url": "https://github.com/derailed/k9s", "title": "K9s: A lazier way to manage Kubernetes Clusters", "date_modified": "2023-07-19T01:18:08.000Z" }, { "content_html": "

While developing with containers is becoming an increasingly popular way for deploying and scaling applications, there are still areas where improvements can be made. One of the main issues with scaling containerized applications is the long startup time, especially during scale up when newer instances need to be added. This issue can have a negative impact on the customer experience, for example when a website needs to scale out to serve additional traffic.

\n

A research paper shows that container image downloads account for 76 percent of container startup time, but on average only 6.4 percent of the data is needed for the container to start doing useful work. Starting and scaling out containerized applications requires downloading container images from a remote container registry. This may introduce a non-trivial latency, as the entire image must be downloaded and unpacked before the applications can be started.

\n

One solution to this problem is lazy loading (also known as asynchronous loading) container images. This approach downloads data from the container registry in parallel with the application startup, such as stargz-snapshotter, a project that aims to improve the overall container start time.

\n

Last year, we introduced Seekable OCI (SOCI), a technology open sourced by Amazon Web Services (AWS) that enables container runtimes to implement lazy loading the container image to start applications faster without modifying the container images. As part of that effort, we open sourced SOCI Snapshotter, a snapshotter plugin that enables lazy loading with SOCI in containerd.

\n

AWS Fargate Support for SOCI
Today, I’m excited to share that AWS Fargate now supports Seekable OCI (SOCI), which helps applications deploy and scale out faster by enabling containers to start without waiting to download the entire container image. At launch, this new capability is available for Amazon Elastic Container Service (Amazon ECS) applications running on AWS Fargate.

\n

Here’s a quick look to show how AWS Fargate support for SOCI works:

\n

\n

SOCI works by creating an index (SOCI index) of the files within an existing container image. This index is a key enabler to launching containers faster, providing the capability to extract an individual file from a container image without having to download the entire image. Your applications no longer need to wait to complete pulling and unpacking a container image before your applications start running. This allows you to deploy and scale out applications more quickly and reduce the rollout time for application updates.

\n

A SOCI index is generated and stored separately from the container images. This means that your container images don’t need to be converted to use SOCI, therefore not breaking secure hash algorithm (SHA)-based security, such as container image signing. The index is then stored in the registry alongside the container image. At release, AWS Fargate support for SOCI works with Amazon Elastic Container Registry (Amazon ECR).

\n

When you use Amazon ECS with AWS Fargate to run your SOCI-indexed containerized images, AWS Fargate automatically detects if a SOCI index for the image exists and starts the container without waiting for the entire image to be pulled. This also means that AWS Fargate will still continue to run container images that don’t have SOCI indexes.

\n

Let’s Get Started
There are two ways to create SOCI indexes for container images.

\n \n

The AWS SOCI Index Builder provides you with an automated process to get started and build SOCI indexes for your container images. The sociCLI provides you with more flexibility around index generation and the ability to natively integrate index generation in your CI/CD pipelines.

\n

In this article, I manually generate SOCI indexes using the soci CLI from the soci-snapshotter project.

\n

Create a Repository and Push Container Images
First, I create an Amazon ECR repository called pytorch-socifor my container image using AWS CLI.

\n
$ aws ecr create-repository --region us-east-1 --repository-name pytorch-soci
\n

I keep the Amazon ECR URI output and define it as a variable to make it easier for me to refer to the repository in the next step.

\n
$ ECRSOCIURI=xyz.dkr.ecr.us-east-1.amazonaws.com/pytorch-soci:latest
\n

For the sample application, I use a PyTorch training (CPU-based) container image from AWS Deep Learning Containers. I use the nerdctl CLI to pull the container image because, by default, the Docker Engine stores the container image in the Docker Engine image store, not the containerd image store.

\n
$ SAMPLE_IMAGE=\"763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.1-cpu-py36-ubuntu16.04\" \n$ aws ecr get-login-password --region us-east-1 | sudo nerdctl login --username AWS --password-stdin xyz.dkr.ecr.ap-southeast-1.amazonaws.com\n$ sudo nerdctl pull --platform linux/amd64 $SAMPLE_IMAGE
\n

Then, I tag the container image for the repository that I created in the previous step.

\n
$ sudo nerdctl tag $SAMPLE_IMAGE $ECRSOCIURI
\n

Next, I need to push the container image into the ECR repository.

\n
$ sudo nerdctl push $ECRSOCIURI
\n

At this point, my container image is already in my Amazon ECR repository.

\n

\n

Create SOCI Indexes
Next, I need to create SOCI index.

\n

A SOCI index is an artifact that enables lazy loading of container images. A SOCI index consists of 1) a SOCI index manifest and 2) a set of zTOCs. The following image illustrates the components in a SOCI index manifest, and how it refers to a container image manifest.

\n

\n

The SOCI index manifest contains the list of zTOCs and a reference to the image for which the manifest was generated. A zTOC, or table of contents for compressed data, consists of two parts:

\n
    \n
  1. TOC, a table of contents containing file metadata and the corresponding offset in the decompressed TAR archive.
    2. \n
  2. zInfo, a collection of checkpoints representing the state of the compression engine at various points in the layer.
    3. \n
\n

To learn more about the concept and term, please visit soci-snapshotter Terminology page.

\n

Before I can create SOCI indexes, I need to install the sociCLI. To learn more about how to install the soci, visit Getting Started with soci-snapshotter.

\n

To create SOCI indexes, I use the soci create command.

\n
$ sudo soci create $ECRSOCIURI\nlayer sha256:4c6ec688ebe374ea7d89ce967576d221a177ebd2c02ca9f053197f954102e30b -> ztoc skipped\nlayer sha256:ab09082b308205f9bf973c4b887132374f34ec64b923deef7e2f7ea1a34c1dad -> ztoc skipped\nlayer sha256:cd413555f0d1643e96fe0d4da7f5ed5e8dc9c6004b0731a0a810acab381d8c61 -> ztoc skipped\nlayer sha256:eee85b8a173b8fde0e319d42ae4adb7990ed2a0ce97ca5563cf85f529879a301 -> ztoc skipped\nlayer sha256:3a1b659108d7aaa52a58355c7f5704fcd6ab1b348ec9b61da925f3c3affa7efc -> ztoc skipped\nlayer sha256:d8f520dcac6d926130409c7b3a8f77aea639642ba1347359aaf81a8b43ce1f99 -> ztoc skipped\nlayer sha256:d75d26599d366ecd2aa1bfa72926948ce821815f89604b6a0a49cfca100570a0 -> ztoc skipped\nlayer sha256:a429d26ed72a85a6588f4b2af0049ae75761dac1bb8ba8017b8830878fb51124 -> ztoc skipped\nlayer sha256:5bebf55933a382e053394e285accaecb1dec9e215a5c7da0b9962a2d09a579bc -> ztoc skipped\nlayer sha256:5dfa26c6b9c9d1ccbcb1eaa65befa376805d9324174ac580ca76fdedc3575f54 -> ztoc skipped\nlayer sha256:0ba7bf18aa406cb7dc372ac732de222b04d1c824ff1705d8900831c3d1361ff5 -> ztoc skipped\nlayer sha256:4007a89234b4f56c03e6831dc220550d2e5fba935d9f5f5bcea64857ac4f4888 -> ztoc sha256:0b4d78c856b7e9e3d507ac6ba64e2e2468997639608ef43c088637f379bb47e4\nlayer sha256:089632f60d8cfe243c5bc355a77401c9a8d2f415d730f00f6f91d44bb96c251b -> ztoc sha256:f6a16d3d07326fe3bddbdb1aab5fbd4e924ec357b4292a6933158cc7cc33605b\nlayer sha256:f18dd99041c3095ade3d5013a61a00eeab8b878ba9be8545c2eabfbca3f3a7f3 -> ztoc sha256:95d7966c964dabb54cb110a1a8373d7b88cfc479336d473f6ba0f275afa629dd\nlayer sha256:69e1edcfbd217582677d4636de8be2a25a24775469d677664c8714ed64f557c3 -> ztoc sha256:ac0e18bd39d398917942c4b87ac75b90240df1e5cb13999869158877b400b865\n
\n

From the above output, I can see that sociCLI created zTOCs for four layers, which and this means only these four layers will be lazily pulled and the other container image layers will be downloaded in full before the container image starts. This is because there is less of a launch time impact in lazy loading very small container image layers. However, you can configure this behavior using the --min-layer-size flag when you run soci create.

\n

Verify and Push SOCI Indexes
The soci CLI also provides several commands that can help you to review the SOCI Indexes that have been generated.

\n

To see a list of all index manifests, I can run the following command.

\n
$ sudo soci index list\n\nDIGEST                                                                     SIZE    IMAGE REF                                                                                   PLATFORM       MEDIA TYPE                                    CREATED\nsha256:ea5c3489622d4e97d4ad5e300c8482c3d30b2be44a12c68779776014b15c5822    1931    xyz.dkr.ecr.us-east-1.amazonaws.com/pytorch-soci:latest                                     linux/amd64    application/vnd.oci.image.manifest.v1+json    10m4s ago\nsha256:ea5c3489622d4e97d4ad5e300c8482c3d30b2be44a12c68779776014b15c5822    1931    763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.1-cpu-py36-ubuntu16.04    linux/amd64    application/vnd.oci.image.manifest.v1+json    10m4s ago\n
\n

While optional, if I need to see the list of zTOC, I can use the following command.

\n
$ sudo soci ztoc list\nDIGEST                                                                     SIZE        LAYER DIGEST\nsha256:0b4d78c856b7e9e3d507ac6ba64e2e2468997639608ef43c088637f379bb47e4    2038072     sha256:4007a89234b4f56c03e6831dc220550d2e5fba935d9f5f5bcea64857ac4f4888\nsha256:95d7966c964dabb54cb110a1a8373d7b88cfc479336d473f6ba0f275afa629dd    11442416    sha256:f18dd99041c3095ade3d5013a61a00eeab8b878ba9be8545c2eabfbca3f3a7f3\nsha256:ac0e18bd39d398917942c4b87ac75b90240df1e5cb13999869158877b400b865    36277264    sha256:69e1edcfbd217582677d4636de8be2a25a24775469d677664c8714ed64f557c3\nsha256:f6a16d3d07326fe3bddbdb1aab5fbd4e924ec357b4292a6933158cc7cc33605b    10152696    sha256:089632f60d8cfe243c5bc355a77401c9a8d2f415d730f00f6f91d44bb96c251b\n
\n

This series of zTOCs contains all of the information that SOCI needs to find a given file in a layer. To review the zTOC for each layer, I can use one of the digest sums from the preceding output and use the following command.

\n
$ sudo soci ztoc info sha256:0b4d78c856b7e9e3d507ac6ba64e2e2468997639608ef43c088637f379bb47e4\n{\n  \"version\": \"0.9\",\n  \"build_tool\": \"AWS SOCI CLI v0.1\",\n  \"size\": 2038072,\n  \"span_size\": 4194304,\n  \"num_spans\": 33,\n  \"num_files\": 5552,\n  \"num_multi_span_files\": 26,\n  \"files\": [\n    {\n      \"filename\": \"bin/\",\n      \"offset\": 512,\n      \"size\": 0,\n      \"type\": \"dir\",\n      \"start_span\": 0,\n      \"end_span\": 0\n    },\n    {\n      \"filename\": \"bin/bash\",\n      \"offset\": 1024,\n      \"size\": 1037528,\n      \"type\": \"reg\",\n      \"start_span\": 0,\n      \"end_span\": 0\n    }\n\n---Trimmed for brevity---\n
\n

Now, I need to use the following command to push all SOCI-related artifacts into the Amazon ECR.

\n
$ PASSWORD=$(aws ecr get-login-password --region us-east-1)\n$ sudo soci push --user AWS:$PASSWORD $ECRSOCIURI
\n

If I go to my Amazon ECR repository, I can verify the index is created. Here, I can see that two additional objects are listed alongside my container image: a SOCI Index and an Image index. The image index allows AWS Fargate to look up SOCI indexes associated with my container image.

\n

\n

Understanding SOCI Performance
The main objective of SOCI is to minimize the required time to start containerized applications. To measure the performance of AWS Fargate lazy loading container images using SOCI, I need to understand how long it takes for my container images to start with SOCI and without SOCI.

\n

To understand the duration needed for each container image to start, I can use metrics available from the DescribeTasks API on Amazon ECS. The first metric is createdAt, the timestamp for the time when the task was created and entered the PENDING state. The second metric is startedAt, the time when the task transitioned from the PENDING state to the RUNNING state.

\n

For this, I have created another Amazon ECR repository using the same container image but without generating a SOCI index, called pytorch-without-soci. If I compare these container images, I have two additional objects in pytorch-soci(an image index and a SOCI index) that don’t exist in pytorch-without-soci.

\n

\n

Deploy and Run Applications
To run the applications, I have created an Amazon ECS cluster called demo-pytorch-soci-cluster, a VPC and the required ECS task execution role. If you’re new to Amazon ECS, you can follow Getting started with Amazon ECS to be more familiar with how to deploy and run your containerized applications.

\n

Now, let’s deploy and run both the container images with FARGATE as the launch type. I define five tasks for each pytorch-sociand pytorch-without-soci.

\n
$ aws ecs \\ \n    --region us-east-1 \\ \n    run-task \\ \n    --count 5 \\ \n    --launch-type FARGATE \\ \n    --task-definition arn:aws:ecs:us-east-1:XYZ:task-definition/pytorch-soci \\ \n    --cluster socidemo \n\n$ aws ecs \\ \n    --region us-east-1 \\ \n    run-task \\ \n    --count 5 \\ \n    --launch-type FARGATE \\ \n    --task-definition arn:aws:ecs:us-east-1:XYZ:task-definition/pytorch-without-soci \\ \n    --cluster socidemo
\n

After a few minutes, there are 10 running tasks on my ECS cluster.

\n

\n

After verifying that all my tasks are running, I run the following script to get two metrics: createdAt and startedAt.

\n
#!/bin/bash\nCLUSTER=<CLUSTER_NAME>\nTASKDEF=<TASK_DEFINITION>\nREGION=\"us-east-1\"\nTASKS=$(aws ecs list-tasks \\\n    --cluster $CLUSTER \\\n    --family $TASKDEF \\\n    --region $REGION \\\n    --query 'taskArns[*]' \\\n    --output text)\n\naws ecs describe-tasks \\\n    --tasks $TASKS \\\n    --region $REGION \\\n    --cluster $CLUSTER \\\n    --query \"tasks[] | reverse(sort_by(@, &createdAt)) | [].[{startedAt: startedAt, createdAt: createdAt, taskArn: taskArn}]\" \\\n    --output table\n
\n

Running the above command for the container image without SOCI indexes — pytorch-without-soci— produces following output:

\n
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n|                                                                                   DescribeTasks                                                                                   |\n+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+\n|             createdAt            |             startedAt             |                                                  taskArn                                                   |\n+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+\n|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:09.856000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/dcdf19b6e66444aeb3bc607a3114fae0   |\n|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:09.459000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/9178b75c98ee4c4e8d9c681ddb26f2ca   |\n|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:21.645000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/7da51e036c414cbab7690409ce08cc99   |\n|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:00.606000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/5ee8f48194874e6dbba75a5ef753cad2   |\n|  2023-07-07T17:43:59.233000+00:00|  2023-07-07T17:46:02.461000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/58531a9e94ed44deb5377fa997caec36   |\n+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+\n
\n

From the average aggregated delta time (between startedAt and createdAt) for each task, the pytorch-without-soci (without SOCI indexes) successfully ran after 129 seconds.

\n

Next, I’m running same command but for pytorch-sociwhich comes with SOCI indexes.

\n
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n|                                                                                   DescribeTasks                                                                                   |\n+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+\n|             createdAt            |             startedAt             |                                                  taskArn                                                   |\n+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+\n|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:51.076000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/c57d8cff6033494b97f6fd0e1b797b8f   |\n|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:52.212000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/6d168f9e99324a59bd6e28de36289456   |\n|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:45:05.443000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/4bdc43b4c1f84f8d9d40dbd1a41645da   |\n|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:50.618000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/43ea53ea84154d5aa90f8fdd7414c6df   |\n|  2023-07-07T17:43:53.318000+00:00|  2023-07-07T17:44:50.777000+00:00 |  arn:aws:ecs:ap-southeast-1:xyz:task/demo-pytorch-soci-cluster/0731bea30d42449e9006a5d8902756d5   |\n+----------------------------------+-----------------------------------+------------------------------------------------------------------------------------------------------------+
\n

Here, I see my container image with SOCI-enabled — pytorch-soci — was started 60 seconds after being created.

\n

This means that running my sample application with SOCI indexes on AWS Fargate is approximately 50 percent faster compared to running without SOCI indexes.

\n

It’s recommended to benchmark the startup and scaling-out time of your application with and without SOCI. This helps you to have a better understanding of how your application behaves and if your applications benefit from AWS Fargate support for SOCI.

\n

Customer Voices
During the private preview period, we heard lots of feedback from our customers about AWS Fargate support for SOCI. Here’s what our customers say:

\n

\n

Autodesk provides critical design, make, and operate software solutions across the architecture, engineering, construction, manufacturing, media, and entertainment industries. “SOCI has given us a 50% improvement in startup performance for our time-sensitive simulation workloads running on Amazon ECS with AWS Fargate. This allows our application to scale out faster, enabling us to quickly serve increased user demand and save on costs by reducing idle compute capacity. The AWS Partner Solution for creating the SOCI index is easy to configure and deploy.” – Boaz Brudner, Head of Innovyze SaaS Engineering, AI and Architecture, Autodesk.

\n

\n

Flywire is a global payments enablement and software company, on a mission to deliver the world’s most important and complex payments. “We run multi-step deployment pipelines on Amazon ECS with AWS Fargate which can take several minutes to complete. With SOCI, the total pipeline duration is reduced by over 50% without making any changes to our applications, or the deployment process. This allowed us to drastically reduce the rollout time for our application updates. For some of our larger images of over 750MB, SOCI improved the task startup time by more than 60%.”, Samuel Burgos, Sr. Cloud Security Engineer, Flywire.

\n

\n

Virtuoso is a leading software corporation that makes functional UI and end-to-end testing software. “SOCI has helped us reduce the lag between demand and availability of compute. We have very bursty workloads which our customers expect to start as fast as possible. SOCI helps our ECS tasks spin-up 40% faster, allowing us to quickly scale our application and reduce the pool of idle compute capacity, enabling us to deliver value more efficiently. Setting up SOCI was really easy. We opted to use the quick-start AWS Partner’s solution with which we could leave our build and deployment pipelines untouched.”, Mathew Hall, Head of Site Reliability Engineering, Virtuoso.

\n

Things to Know
Availability — AWS Fargate support for SOCI is available in all AWS Regions where Amazon ECS, AWS Fargate, and Amazon ECR are available.

\n

Pricing — AWS Fargate support for SOCI is available at no additional cost and you will only be charged for storing the SOCI indexes in Amazon ECR.

\n

Get Started — Learn more about benefits and how to get started on the AWS Fargate Support for SOCI page.

\n

Happy building.
— Donnie

", "url": "https://aws.amazon.com/blogs/aws/aws-fargate-enables-faster-container-startup-using-seekable-oci/", "title": "AWS Fargate Enables Faster Container Startup using Seekable OCI", "date_modified": "2023-07-17T19:29:14.000Z" }, { "content_html": "Comments", "url": "https://katacontainers.io/", "title": "Kata Containers: Virtual Machines that feel and perform like containers", "date_modified": "2023-07-17T12:55:36.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=36736795", "title": "Ask HN: How to price your first Enterprise customer?", "date_modified": "2023-07-15T13:56:55.000Z" }, { "content_html": "Comments", "url": "https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests", "title": "Lies, damned lies: debunking Cloudflare’s recent performance tests", "date_modified": "2023-07-14T23:44:11.000Z" }, { "content_html": "Comments", "url": "https://www.fastly.com/blog/debunking-cloudflares-recent-performance-tests", "title": "Debunking Cloudflare’s recent performance tests (2021)", "date_modified": "2023-07-14T23:44:11.000Z" }, { "content_html": "Comments", "url": "https://github.com/ory/kratos/releases/tag/v1.0.0", "title": "Show HN: Open-source IAM Ory Kratos v1.0 with Passkeys, MFA and multi-region", "date_modified": "2023-07-13T13:37:21.000Z" }, { "content_html": "Comments", "url": "https://digest.browsertech.com/archive/browsertech-digest-figma-is-a-file-editor/", "title": "Figma Is a File Editor", "date_modified": "2023-07-12T23:13:09.000Z" }, { "content_html": "Comments", "url": "https://github.com/cncf/toc/commit/25b2ead46d56e3c39d48a6a87e3c558c2120d179", "title": "Istio moved to CNCF Graduation stage", "date_modified": "2023-07-12T16:57:51.000Z" }, { "content_html": "Comments", "url": "https://www.inngest.com/blog/announcing-inngest-seed-financing", "title": "Inngest raises $3M seed to build the reliable workflow platform for every dev", "date_modified": "2023-07-12T16:11:53.000Z" }, { "content_html": "Comments", "url": "https://commitmono.com/", "title": "Commit Mono – Neutral programming typeface", "date_modified": "2023-07-12T02:25:48.000Z" }, { "content_html": "Comments", "url": "https://neon.tech/blog/http-vs-websockets-for-postgres-queries-at-the-edge", "title": "HTTP vs. WebSockets: Which one is the fastest for Postgres queries at the edge?", "date_modified": "2023-07-11T15:06:42.000Z" }, { "content_html": "

Comments

", "url": "https://metalbear.co/blog/mirrord-as-an-alternative-to-telepresence/", "title": "mirrord as an alternative to Telepresence", "date_modified": "2023-07-11T13:11:52.000Z" }, { "content_html": "Comments", "url": "https://www.suse.com/news/SUSE-Preserves-Choice-in-Enterprise-Linux/", "title": "SUSE is forking RHEL", "date_modified": "2023-07-11T08:41:22.000Z" }, { "content_html": "Comments", "url": "https://qemu.readthedocs.io/en/latest/system/i386/microvm.html", "title": "MicroVM by QEMU", "date_modified": "2023-07-10T22:35:26.000Z" }, { "content_html": "Comments", "url": "https://bpapillon.com/post/feature-flags-theory-vs-reality/", "title": "Feature Flags: Theory vs. Reality", "date_modified": "2023-07-10T15:02:27.000Z" }, { "content_html": "Comments", "url": "https://edgenode.com/", "title": "EdgeNode: Deploy Serverless Containers on Edge", "date_modified": "2023-07-10T14:17:52.000Z" }, { "content_html": "Comments", "url": "https://hocus.dev/blog/qemu-vs-firecracker/", "title": "We replaced Firecracker with QEMU", "date_modified": "2023-07-10T14:15:04.000Z" }, { "content_html": "Comments", "url": "https://community.influxdata.com/t/getting-weird-results-from-gcp-europe-west1/30615", "title": "InfluxDB Cloud shuts down in Belgium; some weren't notified before data deletion", "date_modified": "2023-07-09T19:11:00.000Z" }, { "content_html": "Comments", "url": "https://blog.railway.app/p/introducing-trial-hobby-pro-plans", "title": "The $5 Plan", "date_modified": "2023-07-07T01:40:51.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/announcing-scaler-pro", "title": "PlanetScale Scaler Pro", "date_modified": "2023-07-06T15:23:21.000Z" }, { "content_html": "Comments", "url": "https://www.fastmail.com/blog/partial-outage-on-30-june-2023/", "title": "Fastmail 30 June outage post-mortem", "date_modified": "2023-07-06T14:08:23.000Z" }, { "content_html": "Comments", "url": "https://finance.yahoo.com/news/digitalocean-acquires-paperspace-expand-ai-120000933.html", "title": "DigitalOcean acquires Paperspace (YC W15) for $111M in cash", "date_modified": "2023-07-06T12:45:20.000Z" }, { "content_html": "Comments", "url": "https://arxiv.org/abs/2307.01045", "title": "Cloud Native Software Engineering", "date_modified": "2023-07-05T05:01:06.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/2023-06-25-outage-post-mortem", "title": "June 25th, 2023 Deno Deploy Postmortem", "date_modified": "2023-07-01T11:44:53.000Z" }, { "content_html": "Comments", "url": "https://linear.app/blog/scaling-the-linear-sync-engine", "title": "Scaling Linear's Sync Engine", "date_modified": "2023-06-29T12:23:34.000Z" }, { "content_html": "Comments", "url": "https://blog.jonlu.ca/posts/us-east-1-latency", "title": "Noticing when an app is only hosted in us-east-1", "date_modified": "2023-06-28T19:03:04.000Z" }, { "content_html": "Comments", "url": "https://upvpn.app", "title": "Show HN: Serverless VPN, pay as you go, unlimited devices, no subscriptions", "date_modified": "2023-06-28T18:43:03.000Z" }, { "content_html": "Comments", "url": "https://jmmv.dev/2023/06/fast-machines-slow-machines.html", "title": "Fast machines, slow machines", "date_modified": "2023-06-28T08:53:14.000Z" }, { "content_html": "Comments", "url": "https://cohost.org/tef/post/1764930-how-not-to-write-a", "title": "How not to write a pipeline", "date_modified": "2023-06-28T02:24:00.000Z" }, { "content_html": "Comments", "url": "https://www.docker.com/blog/mutagen-acquisition/", "title": "Docker Acquires Mutagen", "date_modified": "2023-06-27T17:31:52.000Z" }, { "content_html": "Comments", "url": "https://www.influxdata.com/blog/influxdb-3-0-system-architecture/", "title": "InfluxDB 3.0 System Architecture", "date_modified": "2023-06-27T17:17:37.000Z" }, { "content_html": "Comments", "url": "https://github.com/wolfi-dev", "title": "Wolfi: A community Linux OS designed for the container and cloud-native era", "date_modified": "2023-06-27T07:19:37.000Z" }, { "content_html": "Comments", "url": "https://www.winglang.io/blog/2022/11/23/manifesto", "title": "Cloud, Why So Difficult?", "date_modified": "2023-06-27T04:01:04.000Z" }, { "content_html": "Comments", "url": "https://cubxity.dev/blog/folia-test-june-2023", "title": "Testing a 1,000 player Minecraft server with Folia", "date_modified": "2023-06-27T03:13:53.000Z" }, { "content_html": "Comments", "url": "https://akashrajpurohit.com/blog/build-your-own-docker-with-linux-namespaces-cgroups-and-chroot-handson-guide/", "title": "Build Your Own Docker with Linux Namespaces, Cgroups, and Chroot", "date_modified": "2023-06-27T02:50:55.000Z" }, { "content_html": "Comments", "url": "https://determinate.systems/posts/magic-nix-cache", "title": "The Magic Nix Cache, a GitHub Action for speeding up your Nix workflows", "date_modified": "2023-06-26T19:32:38.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=36480230", "title": "Launch HN: Argonaut (YC S21) – Easily Deploy Apps and Infra to AWS and GCP", "date_modified": "2023-06-26T14:41:47.000Z" }, { "content_html": "Comments", "url": "https://spin.atomicobject.com/2023/06/26/dexec-docker/", "title": "Working with Docker Containers Made Easy with the Dexec Bash Script", "date_modified": "2023-06-26T12:29:04.000Z" }, { "content_html": "Comments", "url": "https://blog.warrant.dev/why-zanzibar-shines-at-building-authorization/", "title": "Why Google Zanzibar shines at building authorization", "date_modified": "2023-06-25T17:59:10.000Z" }, { "content_html": "Comments", "url": "https://blog.val.town/blog/introducing-valrun", "title": "Launching val.run: subdomains & arbitrary paths", "date_modified": "2023-06-23T15:56:52.000Z" }, { "content_html": "Comments", "url": "https://changelog.com/posts/webassembly-runtimes-will-replace-container-runtimes-by-2030", "title": "“WebAssembly runtimes will replace container-based runtimes by 2030”", "date_modified": "2023-06-22T16:40:13.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/Articles/935592/", "title": "Red Hat cutting back RHEL source availability", "date_modified": "2023-06-21T16:00:14.000Z" }, { "content_html": "Comments", "url": "https://www.systeminit.com/blog-second-wave-devops/", "title": "System Initiative: Second Wave DevOps", "date_modified": "2023-06-21T15:13:24.000Z" }, { "content_html": "Comments", "url": "https://github.com/juspay/hyperswitch/wiki/Why-I-won%27t-pay-on-your-website", "title": "I won't pay on your website", "date_modified": "2023-06-21T06:33:33.000Z" }, { "content_html": "Comments", "url": "https://render.com/blog/render-series-b", "title": "Render Raises $50M Series B", "date_modified": "2023-06-20T14:58:08.000Z" }, { "content_html": "Comments", "url": "https://www.inngest.com", "title": "Show HN: Inngest – Developer platform for background jobs and workflows", "date_modified": "2023-06-20T12:24:08.000Z" }, { "content_html": "Comments", "url": "https://blog.webp.se/hetzner-arm64-en/", "title": "Review of Hetzner ARM64 Servers and Experience of WebP Cloud Services on Them", "date_modified": "2023-06-17T09:14:22.000Z" }, { "content_html": "Comments", "url": "https://github.com/petykowski/London-Underground-Dot-Matrix-Typeface", "title": "London Underground Dot Matrix Typeface", "date_modified": "2023-06-17T05:52:08.000Z" }, { "content_html": "Comments", "url": "https://llm-utils.org/Nvidia+H100+and+A100+GPUs+-+comparing+available+capacity+at+GPU+cloud+providers", "title": "Nvidia H100 and A100 GPUs – comparing available capacity at GPU cloud providers", "date_modified": "2023-06-14T21:56:21.000Z" }, { "content_html": "Comments", "url": "https://vercel.com/blog/vercel-ai-accelerator", "title": "Vercel's AI Accelerator", "date_modified": "2023-06-14T16:00:03.000Z" }, { "content_html": "Comments", "url": "https://blog.hellolanding.tech/move-fast-avoid-sharp-edges-5d3a07f19a59", "title": "Migrating from Heroku to EKS", "date_modified": "2023-06-14T15:22:01.000Z" }, { "content_html": "Comments", "url": "https://collabfund.com/blog/mental-liquidity/", "title": "Mental Liquidity", "date_modified": "2023-06-11T12:31:34.000Z" }, { "content_html": "Comments", "url": "https://discourse.nixos.org/t/s3-update-and-recap-of-community-call/28942", "title": "Nix/NixOS S3 Update and Recap of Community Call", "date_modified": "2023-06-09T17:56:08.000Z" }, { "content_html": "Comments", "url": "https://old.reddit.com/r/apolloapp/comments/144l6se/apollo_backend_just_made_public_the_goal_of/", "title": "Apollo Back end just made public", "date_modified": "2023-06-09T11:19:58.000Z" }, { "content_html": "
\n
\n \"A\n

Enlarge / A high-res rendered hill inspired by Windows XP's familiar \"Bliss\" wallpaper (visit Microsoft's site to get it at full resolution.) (credit: Microsoft)

\n\n\n\n\n\n\n
\n\n

Did you read the news about the Windows XP activation algorithm getting cracked and suddenly get nostalgic for the blue skies and bluer taskbar of that old Windows release? Or maybe you just like attractive, high-resolution desktop wallpapers and you want to make a change? It turns out that Microsoft's design team has rendered an updated 4K version of the default Windows XP wallpaper—you might know it by its name, \"Bliss.\"

\n

It's one of several retro-themed wallpapers on this Microsoft Design site, including photorealistic renderings of Solitaire, Paint, and (of course) Clippy. The site has been around for a while and hasn't been updated since December 2022, but Windows engineer Jennifer Gentleman tweeted about it yesterday—it's new to me and maybe to you, too. The most recent wallpapers appear to be products of Microsoft's Design Week event.

\n

Among others, the Microsoft Design site also hosts the default wallpapers that have come with several Surface PCs, quite a few Pride Month-themed wallpaper designs, and several images focused on the company's recent emoji redesigns and the icons for the Microsoft 365 apps.

Read 2 remaining paragraphs | Comments

", "url": "https://arstechnica.com/?p=1946449", "title": "Today I stumbled upon Microsoft’s 4K rendering of the Windows XP wallpaper", "date_modified": "2023-06-08T20:22:45.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/davidecci/status/1665835119331135488", "title": "Apple Releases New Static Linker", "date_modified": "2023-06-06T19:59:31.000Z" }, { "content_html": "Comments", "url": "https://spacedimp.com/blog/dockerless-setting-up-an-elixir-webapp-using-podman-and-plug/", "title": "Show HN: Dockerless, Elixir Web Application Using Podman and Plug", "date_modified": "2023-06-06T06:06:26.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudnativefolks.org/ebpf-for-cybersecurity-part-2", "title": "eBPF for Cybersecurity – Part 2", "date_modified": "2023-06-05T14:11:05.000Z" }, { "content_html": "Comments", "url": "https://blog.michael.kuron-germany.de/2023/06/emulating-ppc64-inside-docker/", "title": "Emulating PPC64 Inside Docker", "date_modified": "2023-06-05T02:49:07.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudnativefolks.org/ebpf-for-cybersecurity-part-1", "title": "eBPF for Cybersecurity – Part 1", "date_modified": "2023-06-04T21:36:39.000Z" }, { "content_html": "Comments", "url": "https://github.com/containers/crun", "title": "Crun: Fast and lightweight OCI runtime and C library for running containers", "date_modified": "2023-06-04T20:09:09.000Z" }, { "content_html": "Comments", "url": "https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672", "title": "The NixOS Foundation’s Call to Action: S3 Costs Require Community Support", "date_modified": "2023-06-03T02:10:26.000Z" }, { "content_html": "Comments", "url": "https://mitchellh.com/writing/building-large-technical-projects", "title": "My Approach to Building Large Technical Projects – Mitchell Hashimoto", "date_modified": "2023-06-02T05:32:39.000Z" }, { "content_html": "Comments", "url": "https://www.cnbc.com/2023/06/01/meta-quest-3-unveiled-ahead-of-apples-planned-vr-headset-debut.html", "title": "Zuckerberg unveils Meta’s newest VR headset days before Apple reveals its own", "date_modified": "2023-06-01T14:12:40.000Z" }, { "content_html": "Comments", "url": "https://meltano.com/blog/introducing-meltano-cloud-you-build-the-pipelines-we-manage-the-infrastructure/", "title": "Show HN: Meltano Cloud (GitLab spinout) – Managed infra for open source ELT", "date_modified": "2023-06-01T14:09:25.000Z" }, { "content_html": "Comments", "url": "https://github.com/OAI/moonwalk", "title": "OpenAPI v4 (aka Moonwalk) Proposal", "date_modified": "2023-05-31T21:38:04.000Z" }, { "content_html": "Comments", "url": "https://www.open-fire.dev/", "title": "Show HN: Open Fire Serverless CI", "date_modified": "2023-05-27T21:50:11.000Z" }, { "content_html": "
\n\n\n\n

CI/CD servers are high-value targets for attackers because of their central role in critical development processes. They provide access to source code, a valuable asset for software companies, and can deploy code to production environments, creating serious risks if not adequately secured. Even a single vulnerability can enable attackers to compromise the supply chain, inject malware, and seize control of systems. 

\n\n\n\n

According to “The State of Software Supply Chain Security 2023”, this has led to a rise in supply chain attacks since 2020, and 57% of organizations have suffered security incidents related to DevOps toolchain exposures. 

\n\n\n\n

To avoid data breaches and business disruptions, securing CI/CD servers should be a top priority. Furthermore, Google’s “2022 Accelerate State of DevOps Report” suggests that implementing proper security controls can have a positive impact on software delivery performance.

\n\n\n\n

In this whitepaper, we present 9 effective ways to prevent a supply chain attack on your CI/CD server, providing practical guidance and best practices to help you strengthen security and protect critical development processes.

\n\n\n\n

By implementing these strategies, you can minimize the risk of a supply chain attack and ensure the integrity, availability, and confidentiality of your software supply chain.

\n\n\n\n

\n Download the whitepaper\n

", "url": "https://blog.jetbrains.com/teamcity/2023/05/cicd-security-whitepaper/", "title": "[Whitepaper] 9 Ways to Prevent a Supply Chain Attack on Your CI/CD Server", "date_modified": "2023-05-25T14:07:47.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/v1.34", "title": "Deno 1.34: Deno compile supports NPM packages", "date_modified": "2023-05-25T10:16:00.000Z" }, { "content_html": "Comments", "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9106553/", "title": "Man Cures Acid Reflux By Eating Upside-Down", "date_modified": "2023-05-24T15:12:17.000Z" }, { "content_html": "Comments", "url": "https://www.theregister.com/2023/05/24/windows_365_boot_preview/", "title": "Microsoft enables booting PCs directly into cloud PCs", "date_modified": "2023-05-24T07:11:40.000Z" }, { "content_html": "Comments", "url": "https://developers.redhat.com/articles/2023/05/23/podman-desktop-now-generally-available", "title": "Podman Desktop 1.0", "date_modified": "2023-05-23T15:59:13.000Z" }, { "content_html": "

Comments

", "url": "https://dev.to/livecycle/how-to-host-your-side-projects-for-free-in-2023-from-auth-to-database-42im", "title": "How to host your side-projects for free in 2023: from Auth to Database", "date_modified": "2023-05-22T21:20:52.000Z" }, { "content_html": "

Etrian Odyssey Origins Collection new trailer showcases difficulty options, auto mapping, improved graphics, and more for Switch and Steam.

\n

\n
\n

\nTo read Etrian Odyssey Origins Collection New Trailer Showcases Difficulty Options, Auto Mapping, Improved Graphics, and More in full, please visit The Mako Reactor. Thank you.

", "url": "https://themakoreactor.com/news/etrian-odyssey-origins-collection-new-difficulty-options-gameplay-trailer-nintendo-switch-pc-steam-deck/51158/", "title": "Etrian Odyssey Origins Collection New Trailer Showcases Difficulty Options, Auto Mapping, Improved Graphics, and More", "date_modified": "2023-05-22T13:58:00.000Z" }, { "content_html": "Comments", "url": "https://rosenzweig.io/blog/growing-up-alyssa.html", "title": "Growing Up Alyssa", "date_modified": "2023-05-22T13:09:12.000Z" }, { "content_html": "Comments", "url": "https://brr.fyi/posts/polar-night", "title": "Polar Night", "date_modified": "2023-05-22T09:15:20.000Z" }, { "content_html": "Comments", "url": "https://blog.min.io/erasure-coding-cpu-utilization/", "title": "Benchmarking Intel, AMD and Graviton Using Erasure Coding Workloads", "date_modified": "2023-05-21T13:23:54.000Z" }, { "content_html": "Comments", "url": "https://github.com/leomos/dwgd", "title": "Dwgd: Docker WireGuard Driver", "date_modified": "2023-05-20T20:19:45.000Z" }, { "content_html": "Comments", "url": "https://www.duckware.com/tech/solving-intermittent-cable-modem-issues.html", "title": "Solving Intermittent Cable Modem Issues", "date_modified": "2023-05-20T09:22:21.000Z" }, { "content_html": "Comments", "url": "https://netflixtechblog.com/debugging-a-fuse-deadlock-in-the-linux-kernel-c75cd7989b6d?gi=06e75507c327", "title": "Debugging a FUSE deadlock in the Linux kernel at Netflix", "date_modified": "2023-05-20T06:17:47.000Z" }, { "content_html": "Comments", "url": "https://queue.acm.org/detail.cfm?id=3595878", "title": "Devex: What actually drives productivity", "date_modified": "2023-05-19T20:34:22.000Z" }, { "content_html": "
\"Workers
\"Workers

The Workers Browser Rendering API allows developers to programmatically control and interact with a headless browser instance and create automation flows for their applications and products.

Since the private beta announcement, based on the feedback we've been receiving and our own roadmap, the team has been working on the developer experience and improving the platform architecture for the best possible performance and reliability. Today we enter the open beta and will start onboarding the customers on the wait list.

Developer experience

Starting today, Wrangler, our command-line tool for configuring, building, and deploying applications with Cloudflare developer products, has support for the Browser Rendering API bindings.

You can install Wrangler Beta using npm:

npm install wrangler --save-dev\n
\n

Bindings allow your Workers to interact with resources on the Cloudflare developer platform. In this case, they will provide your Worker script with an authenticated endpoint to interact with a dedicated Chromium browser instance.

This is all you need in your wrangler.toml once this service is enabled for your account:

browser = { binding = \"MYBROWSER\", type = \"browser\" }\n
\n

Now you can deploy any Worker script that requires Browser Rendering capabilities. You can spawn Chromium instances and interact with them programmatically in any way you typically do manually behind your browser.

Under the hood, the Browser Rendering API gives you access to a WebSocket endpoint that speaks the DevTools Protocol. DevTools is what allows us to instrument a Chromium instance running in our global network, and it's the same protocol that Chrome uses on your computer when you inspect a page.

\"Workers

With enough dedication, you can, in fact, implement your own DevTools client and talk the protocol directly. But that'd be crazy; almost no one does that.

So…

Puppeteer

Puppeteer is one of the most popular libraries that abstract the lower-level DevTools protocol from developers and provides a high-level API that you can use to easily instrument Chrome/Chromium and automate browsing sessions. It's widely used for things like creating screenshots, crawling pages, and testing web applications.

Puppeteer typically connects to a local Chrome or Chromium browser using the DevTools port.

We forked a version of Puppeteer and patched it to connect to the Workers Browser Rendering API instead. The changes are minimal; after connecting the developers can then use the full Puppeteer API as they would on a standard setup.

Our version is open sourced here, and the npm can be installed from npmjs as @cloudflare/puppeteer. Using it from a Worker is as easy as:

import puppeteer from \"@cloudflare/puppeteer\";\n
\n

And then all it takes to launch a browser from your script is:

const browser = await puppeteer.launch(env.MYBROWSER);\n
\n

In the long term, we will update Puppeteer to keep matching the version of our Chromium instances infrastructure running in our network.

Developer documentation

Following the tradition with other Developer products, we created a dedicated section for the Browser Rendering APIs in our Developer's Documentation site.

You can access this page to learn more about how the service works, Wrangler support, APIs, and limits, and find examples of starter templates for common applications.

\"Workers

An example application: taking screenshots

Taking screenshots from web pages is one of the typical cases for browser automation.

Let's create a Worker that uses the Browser Rendering API to do just that. This is a perfect example of how to set up everything and get an application running in minutes, it will give you a good overview of the steps involved and the basics of the Puppeteer API, and then you can move from here to other more sophisticated use-cases.

Step one, start a project, install Wrangler and Cloudflare’s fork of Puppeteer:

npm init -f\nnpm install wrangler -save-dev\nnpm install @cloudflare/puppeteer -save-dev\n
\n

Step two, let’s create the simplest possible wrangler.toml configuration file with the Browser Rendering API binding:

name = \"browser-worker\"\nmain = \"src/index.ts\"\ncompatibility_date = \"2023-03-14\"\nnode_compat = true\nworkers_dev = true\n\nbrowser = { binding = \"MYBROWSER\", type = \"browser\" }\n
\n

Step three, create src/index.ts with your Worker code:

import puppeteer from \"@cloudflare/puppeteer\";\n\nexport default {\n    async fetch(request: Request, env: Env): Promise<Response> {\n        const { searchParams } = new URL(request.url);\n        let url = searchParams.get(\"url\");\n        let img: Buffer;\n        if (url) {\n            const browser = await puppeteer.launch(env.MYBROWSER);\n            const page = await browser.newPage();\n            await page.goto(url);\n            img = (await page.screenshot()) as Buffer;\n            await browser.close();\n            return new Response(img, {\n                headers: {\n                    \"content-type\": \"image/jpeg\",\n                },\n            });\n        } else {\n            return new Response(\n                \"Please add the ?url=https://example.com/ parameter\"\n            );\n        }\n    },\n};\n
\n

That's it, no more steps. This Worker instantiates a browser using Puppeteer, opens a new page, navigates to whatever you put in the \"url\" parameter, takes a screenshot of the page, closes the browser, and responds with the JPEG image of the screenshot. It can't get any easier to get started with the Browser Rendering API.

Run npx wrangler dev –remote to test it and npx wrangler publish when you’re done.

\"Workers

You can explore the entire Puppeteer API and implement other functionality and logic from here. And, because it's Workers, you can add other developer products to your code. You might need a relational database, or a KV store to cache your screenshots, or an R2 bucket to archive your crawled pages and assets, or maybe use a Durable Object to keep your browser instance alive and share it with multiple requests, or queues to handle your jobs asynchronous, we have all of this and more.

You can also find this and other examples of how to use Browser Rendering in the Developer Documentation.

How do we use Browser Rendering

Dogfooding our products is one of the best ways to test and improve them, and in some cases, our internal needs dictate or influence our roadmap. Workers Browser Rendering is a good example of that; it was born out of our necessities before we realized it could be a product. We've been using it extensively for things like taking screenshots of pages for social sharing or dashboards, testing web software in CI, or gathering page load performance metrics of our applications.

But there's one product we've been using to stress test and push the limits of the Browser Rendering API and drive the engineering sprints that brought us to open the beta to our customers today: The Cloudflare Radar URL Scanner.

\"Workers

The URL Scanner scans any URL and compiles a full report containing technical, performance, privacy, and security details about that page. It's processing thousands of scans per day currently. It was built on top of Workers and uses a combination of the Browser Rendering APIs with Puppeteer to create enriched HAR archives and page screenshots, Durable Objects to reuse browser instances, Queues to handle customers' load and execute jobs asynchronously, and R2 to store the final reports.

This tool will soon have its own \"how we built it\" blog. Still, we wanted to let you know about it now because it is a good example of how you can build sophisticated applications using Browser Rendering APIs at scale starting today.

Future plans

The team will keep improving the Browser Rendering API, but a few things are worth mentioning today.

First, we are looking into upstreaming the changes in our Puppeteer fork to the main project so that using the official library with the Cloudflare Workers Browser Rendering API becomes as easy as a configuration option.

Second, one of the reasons why we decided to expose the DevTools protocol bare naked in the Worker binding is so that it can support other browser instrumentalization libraries in the future. Playwright is a good example of another popular library that developers want to use.

And last, we are also keeping an eye on and testing WebDriver BiDi, a \"new standard browser automation protocol that bridges the gap between the WebDriver Classic and CDP (DevTools) protocols.\" Click here to know more about the status of WebDriver BiDi.

Final words

The Workers Browser Rendering API enters open beta today. We will gradually be enabling the customers in the wait list in batches and sending them emails. We look forward to seeing what you will be building with it and want to hear from you.

As usual, you can talk to us on our Developers Discord or the Community forum; the team will be listening.

Watch on Cloudflare TV

", "url": "http://blog.cloudflare.com/browser-rendering-open-beta/", "title": "Workers Browser Rendering API enters open beta", "date_modified": "2023-05-19T13:00:32.000Z" }, { "content_html": "\"Announcing

This post is also available in 简体中文, 日本語, Deutsch, Français and Español.

\n
\"Announcing

We’re excited to announce Secrets Store - Cloudflare’s new secrets management offering!

A secrets store does exactly what the name implies - it stores secrets. Secrets are variables that are used by developers that contain sensitive information - information that only authorized users and systems should have access to.

If you’re building an application, there are various types of secrets that you need to manage. Every system should be designed to have identity & authentication data that verifies some form of identity in order to grant access to a system or application. One example of this is API tokens for making read and write requests to a database. Failure to store these tokens securely could lead to unauthorized access of information - intentional or accidental.

The stakes with secret’s management are high. Every gap in the storage of these values has potential to lead to a data leak or compromise. A security administrator’s worst nightmare.

Developers are primarily focused on creating applications, they want to build quickly, they want their system to be performant, and they want it to scale. For them, secrets management is about ease of use, performance, and reliability. On the other hand, security administrators are tasked with ensuring that these secrets remain secure. It’s their responsibility to safeguard sensitive information, ensure that security best practices are met, and to manage any fallout of an incident such as a data leak or breach. It’s their job to verify that developers at their company are building in a secure and foolproof manner.

In order for developers to build at high velocity and for security administrators to feel at ease, companies need to adopt a highly reliable and secure secrets manager. This should be a system that ensures that sensitive information is stored with the highest security measures, while maintaining ease of use that will allow engineering teams to efficiently build.

Why Cloudflare is building a secrets store

Cloudflare’s mission is to help build a better Internet - that means a more secure Internet. We recognize our customers’ need for a secure, centralized repository for storing sensitive data. Within the Cloudflare ecosystem, are various places where customers need to store and access API and authorization tokens, shared secrets, and sensitive information. It’s our job to make it easy for customers to manage these values securely.

The need for secrets management goes beyond Cloudflare. Customers have sensitive data that they manage everywhere - at their cloud provider, on their own infrastructure, across machines. Our plan is to make our Secrets Store a one-stop shop for all of our customer’s secrets.

The evolution of secrets at Cloudflare

In 2020, we launched environment variables and secrets for Cloudflare Workers, allowing customers to create and encrypt variables across their Worker scripts. By doing this, developers can obfuscate the value of a variable so that it’s no longer available in plaintext and can only be accessed by the Worker.

\"Announcing

Adoption and use of these secrets is quickly growing. We now have more than three million Workers scripts that reference variables and secrets managed through Cloudflare. One piece of feedback that we continue to hear from customers is that these secrets are scoped too narrowly.

Today, customers can only use a variable or secret within the Worker that it’s associated with. Instead, customers have secrets that they share across Workers. They don’t want to re-create those secrets and focus their time on keeping them in sync. They want account level secrets that are managed in one place but are referenced across multiple Workers scripts and functions.

Outside of Workers, there are many use cases for secrets across Cloudflare services.

Inside our Web Application Firewall (WAF), customers can make rules that look for authorization headers in order to grant or deny access to requests. Today, when customers create these rules, they put the authorization header value in plaintext, so that anyone with WAF access in the Cloudflare account can see its value. What we’ve heard from our customers is that even internally, engineers should not have access to this type of information. Instead, what our customers want is one place to manage the value of this header or token, so that only authorized users can see, create, and rotate this value. Then when creating a WAF rule, engineers can just reference the associated secret e.g.“account.mysecretauth”. By doing this, we help our customers secure their system by reducing the access scope and enhance management of this value by keeping it updated in one place.

\"Announcing

With new Cloudflare products and features quickly developing, we’re hearing more and more use cases for a centralized secrets manager. One that can be used to store Access Service tokens or shared secrets for Webhooks.

With the new account level Secrets Store, we’re excited to give customers the tools they need to manage secrets across Cloudflare services.

Securing the Secret Store

To have a secrets store, there are a number of measures that need to be in place, and we’re committing to providing these for our customers.

First, we’re going to give the tools that our customers need to restrict access to secrets. We will have scope permissions that will allow admins to choose which users can view, create, edit, or remove secrets. We also plan to add the same level of granularity to our services - giving customers the ability to say “only allow this Worker to access this secret and only allow this set of Firewall rules to access that secret”.

\"Announcing

Next, we’re going to give our customers extensive audits that will allow them to track the access and use of their secrets. Audit logs are crucial for security administrators. They can be used to alert team members that a secret was used by an unauthorized service or that a compromised secret is being accessed when it shouldn’t be. We will give customers audit logs for every secret-related event, so that customers can see exactly who is making changes to secrets and which services are accessing and when.

In addition to the built-in security of the Secrets Store, we’re going to give customers the tools to rotate their encryption keys on-demand or at a cadence that fits the right security posture for them.

Sign up for the beta

We’re excited to get the Secrets Store in our customer’s hands. If you’re interested in using this, please fill out this form, and we’ll reach out to you when it’s ready to use.

Watch on Cloudflare TV

", "url": "http://blog.cloudflare.com/secrets-store/", "title": "Announcing Cloudflare Secrets Store", "date_modified": "2023-05-18T13:00:50.000Z" }, { "content_html": "Comments", "url": "https://smyachenkov.com/posts/book-review-the-staff-engineers-path/", "title": "The Staff Engineer's Path – Book Review", "date_modified": "2023-05-17T13:18:48.000Z" }, { "content_html": "Comments", "url": "https://retrocomputermuseum.co.uk/", "title": "Retro Computer Museum – Leicester, UK", "date_modified": "2023-05-17T12:02:42.000Z" }, { "content_html": "Comments", "url": "https://www.pixelmator.com/blog/2023/05/16/photomator-for-mac-is-out-now/", "title": "Photomator for Mac", "date_modified": "2023-05-17T06:52:08.000Z" }, { "content_html": "Comments", "url": "https://oblivus.com", "title": "Show HN: Oblivus GPU Cloud – Affordable and scalable GPU servers from $0.29/hr", "date_modified": "2023-05-16T07:30:18.000Z" }, { "content_html": "Comments", "url": "https://github.com/brexhq/prompt-engineering", "title": "Brex’s Prompt Engineering Guide", "date_modified": "2023-05-15T00:12:10.000Z" }, { "content_html": "Comments", "url": "https://hpbn.co/", "title": "High Performance Browser Networking", "date_modified": "2023-05-14T12:47:47.000Z" }, { "content_html": "Comments", "url": "https://cyberhost.uk/the-hidden-macos-speedtest-tool-networkquality/", "title": "MacOS networkQuality", "date_modified": "2023-05-14T10:45:08.000Z" }, { "content_html": "Comments", "url": "https://blog.metaobject.com/2023/05/setting-up-hetzner-arm-instances-with.html", "title": "Setting up Hetzner ARM instances with and for Objective-S", "date_modified": "2023-05-14T00:45:06.000Z" }, { "content_html": "Comments", "url": "https://domainverification.org", "title": "Show HN: Automatic Domain Verification", "date_modified": "2023-05-08T17:06:01.000Z" }, { "content_html": "
\n
\n \"Google\n

Enlarge (credit: Aurich Lawson | Getty Images)

\n\n\n\n\n\n\n
\n

By now, you’ve likely heard that passwordless Google accounts have finally arrived. The replacement for passwords is known as \"passkeys.\"

\n

There are many misconceptions about passkeys, both in terms of their usability and the security and privacy benefits they offer compared with current authentication methods. That’s not surprising, given that passwords have been in use for the past 60 years, and passkeys are so new. The long and short of it is that with a few minutes of training, passkeys are easier to use than passwords, and in a matter of months—once a dozen or so industry partners finish rolling out the remaining pieces—using passkeys will be easier still. Passkeys are also vastly more secure and privacy-preserving than passwords, for reasons I'll explain later.

\n

What is a passkey anyway?

\n

This article provides a primer to get people started with Google's implementation of passkeys and explains the technical underpinnings that make them a much easier and more effective way to protect against account takeovers. A handful of smaller sites—specifically, PayPal, Instacart, Best Buy, Kayak, Robinhood, Shop Pay, and Cardpointers—have rolled out various options for logging in with passkeys, but those choices are more proofs of concept than working solutions. Google is the first major online service to make passkeys available, and its offering is refined and comprehensive enough that I’m recommending people turn them on today.

Read 26 remaining paragraphs | Comments

", "url": "https://arstechnica.com/?p=1937113", "title": "Google passkeys are a no-brainer. You’ve turned them on, right?", "date_modified": "2023-05-08T13:50:21.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/TurnerNovak/status/1654577231937544192/photo/1", "title": "A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call", "date_modified": "2023-05-06T02:17:01.000Z" }, { "content_html": "Comments", "url": "https://quii.dev/HTMX_is_the_Future", "title": "Htmx Is the Future", "date_modified": "2023-05-05T14:32:19.000Z" }, { "content_html": "Comments", "url": "https://nextjs.org/blog/next-13-4", "title": "Next.js 13.4", "date_modified": "2023-05-04T15:53:58.000Z" }, { "content_html": "Comments", "url": "https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90", "title": "Prime Video Abandons Serverless Distributed Stack for Monolith, saves 90%", "date_modified": "2023-05-04T06:04:56.000Z" }, { "content_html": "

Comments

", "url": "https://cloud.google.com/architecture/devops", "title": "DevOps capabilities", "date_modified": "2023-05-02T09:09:16.000Z" }, { "content_html": "Comments", "url": "https://service-markup.vercel.app", "title": "Vercel Service Markup", "date_modified": "2023-05-01T22:10:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/Textualize/frogmouth", "title": "Show HN: Frogmouth – A Markdown browser for your terminal", "date_modified": "2023-04-30T13:05:01.000Z" }, { "content_html": "Comments", "url": "https://www.inferless.com/serverless-gpu-market", "title": "The State of Serverless GPUs", "date_modified": "2023-04-28T06:21:27.000Z" }, { "content_html": "Comments", "url": "https://orbstack.dev/", "title": "OrbStack – Fast, lightweight Docker Desktop and Colima alternative for macOS", "date_modified": "2023-04-28T04:21:02.000Z" }, { "content_html": "Comments", "url": "https://anonymoushash.vmbrasseur.com/2023/04/24/software-bill-of-materials-sbom", "title": "An Intro to SBOMs", "date_modified": "2023-04-24T17:01:13.000Z" }, { "content_html": "Comments", "url": "https://glorifiedgluer.com/blog/2023/developing-with-nix-and-make/", "title": "I use Nix and make(1) to develop", "date_modified": "2023-04-24T15:31:05.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=JiuBeLDSGR0", "title": "Why People Aren't Deploying to Vercel Anymore", "date_modified": "2023-04-24T11:04:01.000Z" }, { "content_html": "Comments", "url": "https://philbooth.me/blog/nine-ways-to-shoot-yourself-in-the-foot-with-postgresql", "title": "Ways to shoot yourself in the foot with PostgreSQL", "date_modified": "2023-04-24T06:11:30.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/Articles/926882/", "title": "Standardizing BPF", "date_modified": "2023-04-21T03:19:07.000Z" }, { "content_html": "Comments", "url": "https://evidence.dev", "title": "evidence.dev – Business Intelligence as Code", "date_modified": "2023-04-20T20:01:27.000Z" }, { "content_html": "Comments", "url": "https://github.com/ory/kratos/releases/tag/v0.13.0", "title": "Show HN: Open-source Auth0 alternative Ory Kratos v0.13 released – nearing v1.0", "date_modified": "2023-04-19T14:30:09.000Z" }, { "content_html": "Comments", "url": "https://astral.sh/", "title": "Astral", "date_modified": "2023-04-18T17:38:28.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/pricing-v3/", "title": "Changes to Tailscale Pricing and Plans", "date_modified": "2023-04-18T16:04:41.000Z" }, { "content_html": "Comments", "url": "https://security-blog-prod-wp01.azurewebsites.net/en-us/security/blog/2023/04/06/devops-threat-matrix/", "title": "DevOps Threat Matrix", "date_modified": "2023-04-17T01:44:40.000Z" }, { "content_html": "

Comments

", "url": "https://sethmlarson.dev/google-assured-oss", "title": "Google Assured OSS", "date_modified": "2023-04-14T14:28:30.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/more-throughput/", "title": "Surpassing 10Gb/S over Tailscale", "date_modified": "2023-04-13T18:11:30.000Z" }, { "content_html": "Comments", "url": "https://www.hetzner.com/press-release/arm64-cloud/", "title": "Hetzner Introduces ARM64 Cloud Servers", "date_modified": "2023-04-12T14:17:46.000Z" }, { "content_html": "Comments", "url": "https://tailscale.dev/blog/tailscale-sucks", "title": "Tailscale Sucks", "date_modified": "2023-04-11T19:34:09.000Z" }, { "content_html": "Comments", "url": "https://www.vitling.xyz/toys/swarm/", "title": "Swarm", "date_modified": "2023-04-11T14:30:21.000Z" }, { "content_html": "Comments", "url": "https://supabase.com/blog/edge-runtime-self-hosted-deno-functions", "title": "Supabase Edge Runtime: Self-Hosted Deno Functions", "date_modified": "2023-04-11T13:57:45.000Z" }, { "content_html": "Comments", "url": "https://www.nationalgeographic.com/magazine/article/jesus-tomb-archaeology", "title": "What Archaeology Is Telling Us About the Real Jesus", "date_modified": "2023-04-07T15:34:00.000Z" }, { "content_html": "Comments", "url": "https://engineering.fb.com/2023/04/06/open-source/buck2-open-source-large-scale-build-system/", "title": "Buck2: Our open source build system", "date_modified": "2023-04-06T16:05:51.000Z" }, { "content_html": "Comments", "url": "https://github.com/cilium/pwru", "title": "Packet, where are you? – eBPF-based Linux kernel networking debugger", "date_modified": "2023-04-05T06:11:32.000Z" }, { "content_html": "Comments", "url": "https://www.onefootprint.com/blog/inside-the-enclave-part-2", "title": "Engineering with Enclaves", "date_modified": "2023-04-03T23:36:13.000Z" }, { "content_html": "Comments", "url": "https://github.com/hocus-dev/hocus", "title": "Show HN: Hocus – self-hosted alternative to GitHub Codespaces using Firecracker", "date_modified": "2023-04-03T17:00:42.000Z" }, { "content_html": "

Comments

", "url": "https://tailscale.dev/blog/headscale-funnel", "title": "Using Tailscale without using Tailscale", "date_modified": "2023-04-01T12:47:10.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/announcing-the-planetscale-github-actions", "title": "The PlanetScale GitHub Actions", "date_modified": "2023-03-31T20:38:01.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/tailscale-funnel-beta/", "title": "Tailscale Funnel now available in beta", "date_modified": "2023-03-30T15:31:48.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2023-03-28-introducing-self-service-sboms/", "title": "Self-Service SBOMs", "date_modified": "2023-03-30T12:00:33.000Z" }, { "content_html": "Comments", "url": "https://github.com/frain-dev/convoy", "title": "Show HN: Open-Source Webhooks Gateway for Platform Engineers", "date_modified": "2023-03-30T09:11:39.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/Articles/927262/", "title": "Ubuntu stops shipping Flatpak by default", "date_modified": "2023-03-29T09:37:38.000Z" }, { "content_html": "Comments", "url": "https://ergomake.dev/blog/docker-compose-as-a-universal-interface/", "title": "Docker-compose.yml as a universal infrastructure interface", "date_modified": "2023-03-27T15:12:52.000Z" }, { "content_html": "Comments", "url": "https://blog.mnus.de/2023/03/archlinux-zfs/", "title": "When root on ZFS breaks on Arch Linux", "date_modified": "2023-03-26T12:57:03.000Z" }, { "content_html": "Comments", "url": "https://computer.rip/2023-03-24-docker.html", "title": "Docker", "date_modified": "2023-03-25T21:33:05.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=L_Guz73e6fw", "title": "OpenAI CEO Sam Altman on Lex Fridman Podcast", "date_modified": "2023-03-25T18:31:04.000Z" }, { "content_html": "

Comments

", "url": "https://computer.rip/2023-03-24-docker.html", "title": "Docker, Docker Inc., Docker Hub, and their relation to the broader world of containerization", "date_modified": "2023-03-25T14:43:46.000Z" }, { "content_html": "Comments", "url": "https://podman-desktop.io/downloads", "title": "Podman Desktop: Same functionality as Docker Desktop but open source", "date_modified": "2023-03-24T21:53:57.000Z" }, { "content_html": "Comments", "url": "https://www.docker.com/blog/no-longer-sunsetting-the-free-team-plan/", "title": "Docker: We’re No Longer Sunsetting the Free Team Plan", "date_modified": "2023-03-24T21:26:52.000Z" }, { "content_html": "Comments", "url": "https://taxheaven3000.com/", "title": "Anime dating sim that also does your taxes", "date_modified": "2023-03-24T09:01:39.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/", "title": "We updated our RSA SSH host key", "date_modified": "2023-03-24T05:28:58.000Z" }, { "content_html": "Comments", "url": "https://xdgbasedirectoryspecification.com/", "title": "Use the XDG Base Directory Specification", "date_modified": "2023-03-24T01:37:08.000Z" }, { "content_html": "Comments", "url": "https://fly.io/ruby-dispatch/mrsk-vs-flyio/", "title": "MRSK vs. Fly.io", "date_modified": "2023-03-22T16:56:58.000Z" }, { "content_html": "Comments", "url": "https://dev.37signals.com/bringing-our-apps-back-home/", "title": "De-cloud and de-k8s – bringing our apps back home", "date_modified": "2023-03-22T16:12:13.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/courses/mysql-for-developers/introduction/course-introduction", "title": "MySQL for Developers", "date_modified": "2023-03-21T15:05:11.000Z" }, { "content_html": "Comments", "url": "https://clickhouse.com/blog/building-clickhouse-cloud-from-scratch-in-a-year", "title": "Building ClickHouse Cloud from scratch in a year", "date_modified": "2023-03-20T21:12:42.000Z" }, { "content_html": "Comments", "url": "https://www.infoworld.com/article/3691292/dockers-bad-week.html", "title": "Docker’s Bad Week", "date_modified": "2023-03-20T17:59:12.000Z" }, { "content_html": "Comments", "url": "https://20th.nixos.org/", "title": "20 Years of Nix", "date_modified": "2023-03-18T12:09:59.000Z" }, { "content_html": "Comments", "url": "https://www.warp.dev/blog/introducing-warp-ai", "title": "Warp AI – AI directly integrated into the terminal", "date_modified": "2023-03-16T16:05:44.000Z" }, { "content_html": "Comments", "url": "https://content.citymapper.com/news/2582/citymapper-joins-via", "title": "Citymapper Joins Via", "date_modified": "2023-03-16T14:12:51.000Z" }, { "content_html": "Comments", "url": "https://github.com/chainloop-dev/chainloop", "title": "Show HN: Chainloop, A Software Supply Chain Attestation solution devs won't hate", "date_modified": "2023-03-16T11:34:08.000Z" }, { "content_html": "Comments", "url": "https://status.flyio.net/incidents/sq7fsdlrg92f", "title": "Fly.io outage, recently deployed apps down, deployments disabled", "date_modified": "2023-03-15T20:54:36.000Z" }, { "content_html": "Comments", "url": "https://zed.dev", "title": "Zed, the new code editor from Atom developers, has entered open beta", "date_modified": "2023-03-15T17:23:17.000Z" }, { "content_html": "Comments", "url": "https://modernfontstacks.com/", "title": "Modern Font Stacks", "date_modified": "2023-03-15T14:16:42.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/josevalim/status/1635767176769462272", "title": "Elixir: Docker now charges open source orgs $300", "date_modified": "2023-03-15T11:28:27.000Z" }, { "content_html": "Comments", "url": "https://blog.alexellis.io/docker-is-deleting-open-source-images/", "title": "Docker is deleting Open Source organisations - what you need to know", "date_modified": "2023-03-15T10:57:11.000Z" }, { "content_html": "Comments", "url": "https://github.com/awslabs/mountpoint-s3", "title": "Mountpoint – file client for S3 written in Rust, from AWS", "date_modified": "2023-03-14T18:12:30.000Z" }, { "content_html": "Comments", "url": "https://modernfontstacks.com/", "title": "Show HN: Modern Font Stacks – New system font stack CSS for modern OSs", "date_modified": "2023-03-14T12:24:48.000Z" }, { "content_html": "Comments", "url": "https://discuss.systems/@dan/110008052977994607", "title": "Audiophile forum debating which versions of memcpy had the highest sound quality (2013)", "date_modified": "2023-03-13T15:44:54.000Z" }, { "content_html": "Comments", "url": "https://nickdesaulniers.github.io/blog/2023/03/10/disambiguating-arm/", "title": "Disambiguating Arm, Arm ARM, ARMv9, ARM9, ARM64, AArch64, A64, A78, ...", "date_modified": "2023-03-10T20:16:57.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/introducing-oxy/", "title": "Oxy is Cloudflare's Rust-based next generation proxy framework", "date_modified": "2023-03-10T16:35:17.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=35096366", "title": "Launch HN: Defer (YC W23) – Zero-infrastructure background jobs for Node.js", "date_modified": "2023-03-10T16:16:28.000Z" }, { "content_html": "Comments", "url": "https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-by-not-going-to-the-cloud-8939dd930af8?gi=16d65c893933", "title": "Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud", "date_modified": "2023-03-10T14:04:19.000Z" }, { "content_html": "Comments", "url": "https://fluxkeyboard.com/", "title": "Flux Keyboard", "date_modified": "2023-03-09T17:15:03.000Z" }, { "content_html": "Comments", "url": "https://blog.cachix.org/posts/2023-03-08-cachix-1-3-uploads-unleashed/", "title": "Cachix 1.3: Nix uploads unleashed", "date_modified": "2023-03-08T10:12:00.000Z" }, { "content_html": "Comments", "url": "https://blog.cachix.org/posts/2023-03-08-cachix-1-3-uploads-unleashed/", "title": "Cachix 1.3: Uploads unleashed", "date_modified": "2023-03-08T10:12:00.000Z" }, { "content_html": "Comments", "url": "https://coutant.org/1.html", "title": "Microphones", "date_modified": "2023-03-08T09:23:59.000Z" }, { "content_html": "Comments", "url": "https://dagster.io/blog/fast-deploys-with-pex-and-docker", "title": "How we deploy faster with warm Docker containers", "date_modified": "2023-03-08T01:42:39.000Z" }, { "content_html": "Comments", "url": "https://earthly.dev/blog/remote-code-execution/", "title": "Remote Code Execution as a Service", "date_modified": "2023-03-07T19:29:49.000Z" }, { "content_html": "Comments", "url": "https://www.richardtaylor.dev/articles/globally-distributed-elixir-over-tailscale", "title": "Globally Distributed Elixir over Tailscale", "date_modified": "2023-03-07T19:12:49.000Z" }, { "content_html": "Comments", "url": "https://vercel.com/blog/framework-defined-infrastructure", "title": "Framework-Defined Infrastructure", "date_modified": "2023-03-07T16:17:04.000Z" }, { "content_html": "Comments", "url": "https://matthewgrohman.substack.com/p/want-an-unfair-advantage-in-your", "title": "Want an unfair advantage in your tech career? Consume content for other roles", "date_modified": "2023-03-06T12:27:06.000Z" }, { "content_html": "Comments", "url": "https://www.githubstatus.com/incidents/bypf7bqdh4g0", "title": "GitHub Packages Is Down", "date_modified": "2023-03-01T13:07:29.000Z" }, { "content_html": "

Recently I started using a Mac for the first time. The biggest downside I’ve\nnoticed so far is that the package management is much worse than on Linux.\nAt some point I got frustrated with homebrew because I felt like it was\nspending too much time upgrading when I installed new packages, and so I\nthought – maybe I’ll try the nix package manager!

\n\n

nix has a reputation for being confusing (it has its whole\nown programming language!), so I’ve been trying to figure out how to use nix in\na way that’s as simple as possible and does not involve managing any\nconfiguration files or learning a new programming language. Here’s what I’ve\nfigured out so far! We’ll talk about how to:

\n\n
    \n
  1. install packages with nix
    2. \n
  2. build a custom nix package for a C++ program called paperjam
    3. \n
  3. install a 5-year-old version of hugo with nix
    4. \n
\n\n

As usual I’ve probably gotten some stuff wrong in this post since I’m still\npretty new to nix. I’m also still not sure how much I like nix – it’s very\nconfusing! But it’s helped me compile some software that I was struggling to\ncompile otherwise, and in general it seems to install things faster than\nhomebrew.

\n\n

what’s interesting about nix?

\n\n

People often describe nix as “declarative package management”. I don’t\ncare that much about declarative package management, so here are two things\nthat I appreciate about nix:

\n\n
    \n
  1. It provides binary packages (hosted at cache.nixos.org/) that you can quickly download and install
    2. \n
  2. For packages which don’t have binary packages, it makes it easier to compile them
    3. \n
\n\n

I think that the reason nix is good at compiling software is that:

\n\n\n\n

I’ll give a couple of examples later in this post of two times nix made it easier for me to compile software.

\n\n

how I got started with nix

\n\n

here’s how I got started with nix:

\n\n
    \n
  1. Install nix. I forget exactly how I did this, but it looks like there’s an official installer and an unofficial installer from zero-to-nix.com. The instructions for uninstalling nix on MacOS with the standard multi-user install are a bit complicated, so it might be worth choosing an installation method with simpler uninstall instructions.
    2. \n
  2. Put ~/.nix-profile/bin on my PATH
    3. \n
  3. Install packages with nix-env -iA nixpkgs.NAME
    4. \n
  4. That’s it.
    5. \n
\n\n

Basically the idea is to treat nix-env -iA like brew install or apt-get install.

\n\n

For example, if I want to install fish, I can do that like this:

\n\n
nix-env -iA nixpkgs.fish\n
\n\n

This seems to just download some binaries from cache.nixos.org – pretty simple.

\n\n

Some people use nix to install their Node and Python and Ruby packages, but I haven’t\nbeen doing that – I just use npm install and pip install the same way I\nalways have.

\n\n

some nix features I’m not using

\n\n

There are a bunch of nix features/tools that I’m not using, but that I’ll\nmention. I originally thought that you had to use these features to use nix,\nbecause most of the nix tutorials I’ve read talk about them. But you don’t have to use them.

\n\n\n\n

I won’t go into these because I haven’t really used them and there are lots of\nexplanations out there.

\n\n

where are nix packages defined?

\n\n

I think packages in the main nix package repository are defined in github.com/NixOS/nixpkgs/

\n\n

It looks like you can search for packages at search.nixos.org/packages. The two official ways to search packages seem to be:

\n\n\n\n

I found a way to search nix packages from the command line that I liked better:

\n\n
    \n
  1. Run nix-env -qa '*' > nix-packages.txt to get a list of every package in the Nix repository
    2. \n
  2. Write a short nix-search script that just greps packages.txt (cat ~/bin/nix-packages.txt | awk '{print $1}' | rg \"$1\")
    3. \n
\n\n

everything is installed with symlinks

\n\n

One of nix’s major design choices is that there isn’t one single bin with all\nyour packages, instead you use symlinks. There are a lot of layers of symlinks. A few examples of symlinks:

\n\n\n\n

When I install something, it creates a new profile-112-link directory with new symlinks and updates my ~/.nix-profile to point to that directory.

\n\n

I think this means that if I install a new version of fish and I don’t like it, I can\neasily go back just by running nix-env --rollback – it’ll move me to my previous profile directory.

\n\n

uninstalling packages doesn’t delete them

\n\n

If I uninstall a nix package like this, it doesn’t actually free any hard drive space, it just removes the symlinks.

\n\n
$ nix-env --uninstall oil\n
\n\n

I’m still not sure how to actually delete the package – I ran a garbage collection like this, which seemed to delete some things:

\n\n
$ nix-collect-garbage\n...\n85 store paths deleted, 74.90 MiB freed\n
\n\n

But I still have oil on my system at /nix/store/8pjnk6jr54z77jiq5g2dbx8887dnxbda-oil-0.14.0.

\n\n

There’s a more aggressive version of nix-collect-garbage that also deletes old versions of your profiles (so that you can’t rollback)

\n\n
$ nix-collect-garbage -d --delete-old\n
\n\n

That doesn’t delete /nix/store/8pjnk6jr54z77jiq5g2dbx8887dnxbda-oil-0.14.0 either though and I’m not sure why.

\n\n

upgrading

\n\n

It looks like you can upgrade nix packages like this:

\n\n
nix-channel --update\nnix-env --upgrade\n
\n\n

(similar to apt-get update && apt-get upgrade)

\n\n

I haven’t really upgraded anything yet. I think that if something goes wrong with an upgrade, you can roll back (because everything is immutable in nix!) with

\n\n
nix-env --rollback\n
\n\n

Someone linked me to this post from Ian Henry that\ntalks about some confusing problems with nix-env --upgrade – maybe it\ndoesn’t work the way you’d expect? I guess I’ll be wary around upgrades.

\n\n

next goal: make a custom package of paperjam

\n\n

After a few months of installing existing packages, I wanted to make a custom package with nix for a program called paperjam that wasn’t already packaged.

\n\n

I was actually struggling to compile paperjam at all even without nix because the version I had\nof libiconv I has on my system was wrong. I thought it might be easier to\ncompile it with nix even though I didn’t know how to make nix packages yet. And\nit actually was!

\n\n

But figuring out how to get there was VERY confusing, so here are some notes about how I did it.

\n\n

how to build an example package

\n\n

Before I started working on my paperjam package, I wanted to build an example existing package just to\nmake sure I understood the process for building a package. I was really\nstruggling to figure out how to do this, but I asked in Discord and someone\nexplained to me how I could get a working package from github.com/NixOS/nixpkgs/ and build it. So here\nare those instructions:

\n\n

step 1: Download some arbitrary package from nixpkgs on github, for example the dash package:

\n\n
wget https://raw.githubusercontent.com/NixOS/nixpkgs/47993510dcb7713a29591517cb6ce682cc40f0ca/pkgs/shells/dash/default.nix -O dash.nix\n
\n\n

step 2: Replace the first statement ({ lib , stdenv , buildPackages , autoreconfHook , pkg-config , fetchurl , fetchpatch , libedit , runCommand , dash }: with with import <nixpkgs> {}; I don’t know why you have to do this,\nbut it works.

\n\n

step 3: Run nix-build dash.nix

\n\n

This compiles the package

\n\n

step 4: Run nix-env -i -f dash.nix

\n\n

This installs the package into my ~/.nix-profile

\n\n

That’s all! Once I’d done that, I felt like I could modify the dash package and make my own package.

\n\n

how I made my own package

\n\n

paperjam has one dependency (libpaper) that also isn’t packaged yet, so I needed to build libpaper first.

\n\n

Here’s libpaper.nix. I basically just wrote this by copying and pasting from\nother packages in the nixpkgs repository.\nMy guess is what’s happening here is that nix has some default rules for\ncompiling C packages (like “run make install”), so the make install happens\ndefault and I don’t need to configure it explicitly.

\n\n
with import <nixpkgs> {};\n\nstdenv.mkDerivation rec {\n  pname = \"libpaper\";\n  version = \"0.1\";\n\n  src = fetchFromGitHub {\n    owner = \"naota\";\n    repo = \"libpaper\";\n    rev = \"51ca11ec543f2828672d15e4e77b92619b497ccd\";\n    hash = \"sha256-S1pzVQ/ceNsx0vGmzdDWw2TjPVLiRgzR4edFblWsekY=\";\n  };\n\n  buildInputs = [ ];\n\n  meta = with lib; {\n    homepage = \"https://github.com/naota/libpaper\";\n    description = \"libpaper\";\n    platforms = platforms.unix;\n    license = with licenses; [ bsd3 gpl2 ];\n  };\n}\n
\n\n

Basically this just tells nix how to download the source from GitHub.

\n\n

I built this by running nix-build libpaper.nix

\n\n

Next, I needed to compile paperjam. Here’s a link to the nix package I wrote. The main things I needed to do other than telling it where to download the source were:

\n\n
    \n
  1. add some extra build dependencies (like asciidoc)
    2. \n
  2. set some environment variables for the install (installFlags = [ \"PREFIX=$(out)\" ];) so that it installed in the correct directory instead of /usr/local/bin.
    3. \n
\n\n

I set the hashes by first leaving the hash empty, then running nix-build to get an error message complaining about a mismatched hash. Then I copied the correct hash out of the error message.

\n\n

I figured out how to set installFlags just by running rg PREFIX\nin the nixpkgs repository – I figured that needing to set a PREFIX was\npretty common and someone had probably done it before, and I was right. So I\njust copied and pasted that line from another package.

\n\n

Then I ran:

\n\n
nix-build paperjam.nix\nnix-env -i -f paperjam.nix\n
\n\n

and then everything worked and I had paperjam installed! Hooray!

\n\n

next goal: install a 5-year-old version of hugo

\n\n

Right now I build this blog using Hugo 0.40, from 2018. I don’t need any new\nfeatures so I haven’t felt a need to upgrade. On Linux this is easy: Hugo’s\nreleases are a static binary, so I can just download the 5-year-old binary from\nthe releases page and\nrun it. Easy!

\n\n

But on this Mac I ran into some complications. Mac hardware has changed in the\nlast 5 years, so the Mac Hugo binary I downloaded crashed. And when I tried to\nbuild it from source with go build, that didn’t work either because Go build\nnorms have changed in the last 5 years as well.

\n\n

I was working around this by running Hugo in a Linux docker container, but I\ndidn’t love that: it was kind of slow and it felt silly. It shouldn’t be that\nhard to compile one Go program!

\n\n

Nix to the rescue! Here’s what I did to install the old version of Hugo with\nnix.

\n\n

installing Hugo 0.40 with nix

\n\n

I wanted to install Hugo 0.40 and put it in my PATH as hugo-0.40. Here’s how\nI did it. I did this in a kind of weird way, but it worked (Searching and installing old versions of Nix packages\ndescribes a probably more normal method).

\n\n

step 1: Search through the nixpkgs repo to find Hugo 0.40

\n\n

I found the .nix file here github.com/NixOS/nixpkgs/blob/17b2ef2/p…

\n\n

step 2: Download that file and build it

\n\n

I downloaded that file (and another file called deps.nix in the same directory), replaced the first line with with import <nixpkgs> {};, and built it with nix-build hugo.nix.

\n\n

That almost worked without any changes, but I had to make two changes:

\n\n\n\n

step 3: Rename hugo to hugo-0.40

\n\n

I write a little post install script to rename the Hugo binary.

\n\n
  postInstall = ''\n    mv $out/bin/hugo $out/bin/hugo-0.40\n  '';\n
\n\n

I figured out how to run this by running rg 'mv ' in the nixpkgs repository and just copying and modifying something that seemed related.

\n\n

step 4: Install it

\n\n

I installed into my ~/.nix-profile/bin by running nix-env -i -f hugo.nix.

\n\n

And it all works! I put the final .nix file into my own personal nixpkgs repo so that I can use it again later if I\nwant.

\n\n

reproducible builds aren’t magic, they’re really hard

\n\n

I think it’s worth noting here that this hugo.nix file isn’t magic – the\nreason I can easily compile Hugo 0.40 today is that many people worked for a long time to make it possible to\npackage that version of Hugo in a reproducible way.

\n\n

that’s all!

\n\n

Installing paperjam and this 5-year-old version of Hugo were both\nsurprisingly painless and actually much easier than compiling it without nix,\nbecause nix made it much easier for me to compile the paperjam package with\nthe right version of libiconv, and because someone 5 years ago had already\ngone to the trouble of listing out the exact dependencies for Hugo.

\n\n

I don’t have any plans to get much more complicated with nix (and it’s still\nvery possible I’ll get frustrated with it and go back to homebrew!), but we’ll\nsee what happens! I’ve found it much easier to start in a simple way and then\nstart using more features if I feel the need instead of adopting a whole bunch\nof complicated stuff all at once.

\n\n

I probably won’t use nix on Linux – I’ve always been happy enough with apt\n(on Debian-based distros) and pacman (on Arch-based distros), and they’re\nmuch less confusing. But on a Mac it seems like it might be worth it. We’ll\nsee! It’s very possible in 3 months I’ll get frustrated with nix and just go back to homebrew.

\n\n

5-month update: rebuilding my nix profile

\n\n

Update from 5 months in: nix is still going well, and I’ve only run into 1\nproblem, which is that every nix-env -iA package installation started failing\nwith the error “bad meta.outputsToInstall”.

\n\n

This script\nfrom Ross Light fixes that problem though. It lists every derivation installed\nin my current profile and creates a new profile with the exact same\nderivations. This feels like a nix bug (surely creating a new profile with the\nexact same derivations should be a no-op?) but I haven’t looked into it more yet.

", "url": "https://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/", "title": "Some notes on using nix", "date_modified": "2023-02-28T23:16:17.000Z" }, { "content_html": "Comments", "url": "https://benhoyt.com/writings/flyio/", "title": "From Go on EC2 to Fly.io", "date_modified": "2023-02-27T05:13:09.000Z" }, { "content_html": "Comments", "url": "https://blog.railway.app/p/scaling-railway-automating-support", "title": "Serving 250k Developers with One Support Engineer", "date_modified": "2023-02-24T20:22:46.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/v1.31", "title": "Deno 1.31: Package.json Support", "date_modified": "2023-02-24T15:16:37.000Z" }, { "content_html": "Comments", "url": "https://www.psypost.org/2023/02/camouflaging-of-autistic-traits-linked-to-internalizing-symptoms-such-as-anxiety-and-depression-68382", "title": "Camouflaging autistic traits linked to internalizing symptoms anxiety depression", "date_modified": "2023-02-23T17:05:29.000Z" }, { "content_html": "Comments", "url": "https://www.vice.com/en/article/y3py9j/ai-companion-replika-erotic-roleplay-updates", "title": "AI Companion Users Are in Crisis, Reporting Sudden Sexual Rejection", "date_modified": "2023-02-22T17:37:30.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34898253", "title": "Launch HN: Depot (YC W23) – Fast Docker Builds in the Cloud", "date_modified": "2023-02-22T16:40:15.000Z" }, { "content_html": "Comments", "url": "https://github.com/highlight/highlight", "title": "Show HN: We’re open-sourcing our session replay tool", "date_modified": "2023-02-22T16:02:26.000Z" }, { "content_html": "Comments", "url": "https://techcrunch.com/2023/02/21/soylent-acquired-starco-brands-nutrition/", "title": "Soylent Acquired by Starco Brands", "date_modified": "2023-02-22T05:02:09.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34885077", "title": "Launch HN: Moonrepo (YC W23) – Open-source build system", "date_modified": "2023-02-21T18:48:10.000Z" }, { "content_html": "Comments", "url": "https://www.fastly.com/blog/announcing-certainly-fastlys-own-tls-certification-authority", "title": "Certainly: Fastly’s Own TLS Certification Authority", "date_modified": "2023-02-21T15:13:43.000Z" }, { "content_html": "Comments", "url": "https://goteleport.com/blog/passkeys/", "title": "Passkeys for Infrastructure", "date_modified": "2023-02-21T15:06:27.000Z" }, { "content_html": "Comments", "url": "https://github.com/containers/podman/releases/tag/v4.4.0", "title": "Podman 4.4", "date_modified": "2023-02-20T22:55:04.000Z" }, { "content_html": "Comments", "url": "https://www.1place.app/blog/the-saas-2-0-manifesto", "title": "Show HN: The SaaS 2.0 Manifesto", "date_modified": "2023-02-20T17:46:19.000Z" }, { "content_html": "Comments", "url": "https://about.gitlab.com/handbook/acquisitions/acquisition-process/", "title": "Gitlab's Startup Acquisition Process", "date_modified": "2023-02-20T10:32:35.000Z" }, { "content_html": "Comments", "url": "https://blog.newsblur.com/2021/06/28/story-of-a-hacking/", "title": "A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)", "date_modified": "2023-02-19T18:16:23.000Z" }, { "content_html": "Comments", "url": "https://www.tinybird.co/blog-posts/better-ci-pipeline-with-data", "title": "We cut our CI pipeline execution time in half", "date_modified": "2023-02-17T15:50:53.000Z" }, { "content_html": "Comments", "url": "https://github.com/BretFisher/awesome-swarm", "title": "Bret Fisher – Awesome-Swarm", "date_modified": "2023-02-16T20:01:00.000Z" }, { "content_html": "Comments", "url": "https://brew.sh/2023/02/16/homebrew-4.0.0/", "title": "Homebrew 4.0.0", "date_modified": "2023-02-16T11:00:57.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/BigscreenVR/status/1625152589624135698", "title": "Bigscreen VR announces Beyond: the world's smallest VR headset", "date_modified": "2023-02-13T15:52:32.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/kelseyhightower/status/1624081136073994240", "title": "Databases on Kubernetes is fundamentally same as a database on a VM", "date_modified": "2023-02-11T08:03:58.000Z" }, { "content_html": "Comments", "url": "https://status.flyio.net", "title": "Ask HN: Are people considering moving off of Fly.io?", "date_modified": "2023-02-10T17:39:10.000Z" }, { "content_html": "Comments", "url": "https://www.linode.com/docs/guides/podman-vs-docker/", "title": "Podman vs. Docker: Comparing the two containerization tools", "date_modified": "2023-02-09T02:46:38.000Z" }, { "content_html": "
\n
\n \"Hand\n

Enlarge (credit: OnePlus)

\n\n\n\n\n\n\n
\n

OnePlus is finally ready to detail its first mechanical keyboard. No, we didn't need another company to start making mechanical keyboards. But if you're looking for a new Bluetooth keyboard that plays particularly well with Macs, has a compact layout, and a rotary knob that looks stylish and functional, OnePlus will have one more choice for you come April.

\n

Announced today, OnePlus is jumping into the mechanical keyboard race with a strange name, the Featuring Keyboard 81 Pro. The \"81\" refers to the key count, while \"Pro\" is assumably meant to make workers and power users think the keyboard's a good fit; but the name doesn't quite roll off the tongue. The outlier here is the \"Featuring\" bit, which refers to the OnePlus Featuring \"co-creation\" platform that builds products based off user feedback. Community users are said to have contributed to the 81 Pro's design, including its proprietary switches. OnePlus' press release today claimed it will release \"many\" more Featuring products.

\n
\"OnePlus-y

OnePlus-y touches include a logo-stamped Esc key. (credit: OnePlus/Twitter)

\n

Another huge influence on the 81 Pro is keyboard-maker Keychron, which is said to have helped engineer the product. That includes its layout, which matches the layout of the Q1 Pro that Keychron is currently crowdfunding. In addition to macOS, the keyboard is supposed to work with Windows, Linux, and Android, OnePlus' press release said. The keyboard's product page also claims support with iOS. Similar to some wireless Keychron keyboards, like the Keychron K14, there's a toggle on the keyboard's side for switching from Mac to Windows. Considering the lack of USB-A ports among Macs, the Bluetooth 5.1 keyboard charges over a USB-C to USB-C cable (there's also a USB-C to USB-A adapter).

Read 7 remaining paragraphs | Comments

", "url": "https://arstechnica.com/?p=1915672", "title": "OnePlus unveils its first mechanical keyboard: Mac layout, custom switches", "date_modified": "2023-02-07T19:10:42.000Z" }, { "content_html": "Comments", "url": "https://www.theverge.com/2023/2/7/23587454/microsoft-bing-edge-chatgpt-ai", "title": "Microsoft announces new Bing and Edge browser powered by upgraded ChatGPT AI", "date_modified": "2023-02-07T18:27:42.000Z" }, { "content_html": "
\"How
\"How

Over the years when Cloudflare has had an outage that affected our customers we have very quickly blogged about what happened, why, and what we are doing to address the causes of the outage. Today’s post is a little different. It’s about a single customer’s website not working correctly because of incorrect action taken by Cloudflare.

Although the customer was not in any way banned from Cloudflare, or lost access to their account, their website didn’t work. And it didn’t work because Cloudflare applied a bandwidth throttle between us and their origin server. The effect was that the website was unusable.

Because of this unusual throttle there was some internal confusion for our customer support team about what had happened. They, incorrectly, believed that the customer had been limited because of a breach of section 2.8 of our Self-Serve Subscription Agreement which prohibits use of our self-service CDN to serve excessive non-HTML content, such as images and video, without a paid plan that includes those services (this is, for example, designed to prevent someone building an image-hosting service on Cloudflare and consuming a huge amount of bandwidth; for that sort of use case we have paid image and video plans).

However, this customer wasn’t breaking section 2.8, and they were both a paying customer and a paying customer of Cloudflare Workers through which the throttled traffic was passing. This throttle should not have happened. In addition, there is and was no need for the customer to upgrade to some other plan level.

This incident has set off a number of workstreams inside Cloudflare to ensure better communication between teams, prevent such an incident happening, and to ensure that communications between Cloudflare and our customers are much clearer.

Before we explain our own mistake and how it came to be, we’d like to apologize to the customer. We realize the serious impact this had, and how we fell short of expectations. In this blog post, we want to explain what happened, and more importantly what we’re going to change to make sure it does not happen again.

Background

On February 2, an on-call network engineer received an alert for a congesting interface with Equinix IX in our Ashburn data center. While this is not an unusual alert, this one stood out for two reasons. First, it was the second day in a row that it happened, and second, the congestion was due to a sudden and extreme spike of traffic.

\"How

The engineer in charge identified the customer’s domain, tardis.dev, as being responsible for this sudden spike of traffic between Cloudflare and their origin network, a storage provider. Because this congestion happens on a physical interface connected to external peers, there was an immediate impact to many of our customers and peers. A port congestion like this one typically incurs packet loss, slow throughput and higher than usual latency. While we have automatic mitigation in place for congesting interfaces, in this case the mitigation was unable to resolve the impact completely.

The traffic from this customer went suddenly from an average of 1,500 requests per second, and a 0.5 MB payload per request, to 3,000 requests per second (2x) and more than 12 MB payload per request (25x).

\"How

The congestion happened between Cloudflare and the origin network. Caching did not happen because the requests were all unique URLs going to the origin, and therefore we had no ability to serve from cache.

A Cloudflare engineer decided to apply a throttling mechanism to prevent the zone from pulling so much traffic from their origin. Let's be very clear on this action: Cloudflare does not have an established process to throttle customers that consume large amounts of bandwidth, and does not intend to have one. This remediation was a mistake, it was not sanctioned, and we deeply regret it.

We lifted the throttle through internal escalation 12 hours and 53 minutes after having set it up.

What's next

To make sure a similar incident does not happen, we are establishing clear rules to mitigate issues like this one. Any action taken against a customer domain, paying or not, will require multiple levels of approval and clear communication to the customer. Our tooling will be improved to reflect this. We have many ways of traffic shaping in situations where a huge spike of traffic affects a link and could have applied a different mitigation in this instance.

We are in the process of rewriting our terms of service to better reflect the type of services that our customers deliver on our platform today. We are also committed to explaining to our users in plain language what is permitted under self-service plans. As a developer-first company with transparency as one of its core principles, we know we can do better here. We will follow up with a blog post dedicated to these changes later.

Once again, we apologize to the customer for this action and for the confusion it created for other Cloudflare customers.

", "url": "https://blog.cloudflare.com/how-cloudflare-erroneously-throttled-a-customers-web-traffic/", "title": "How Cloudflare erroneously throttled a customer’s web traffic", "date_modified": "2023-02-07T18:20:49.000Z" }, { "content_html": "Comments", "url": "https://floxdev.com/blog/flox-open-beta", "title": "The Flox Open Beta", "date_modified": "2023-02-07T14:50:26.000Z" }, { "content_html": "Comments", "url": "https://www.vcinsights.co/", "title": "Show HN: Database of every VC investment memo", "date_modified": "2023-02-07T13:56:24.000Z" }, { "content_html": "Comments", "url": "https://ochagavia.nl/blog/crafting-container-images-without-dockerfiles/", "title": "Crafting container images without Dockerfiles", "date_modified": "2023-02-06T14:54:32.000Z" }, { "content_html": "Comments", "url": "https://protocol.ai/blog/2023-02-03-crypto-winter-update/", "title": "Protocol Labs is laying off 21% of staff (89 people)", "date_modified": "2023-02-04T14:07:19.000Z" }, { "content_html": "Comments", "url": "https://webapp.io/hosting", "title": "Show HN: Webapp.io - Free firecracker-based full-stack hosting", "date_modified": "2023-02-03T22:35:25.000Z" }, { "content_html": "

So you’ve played, or re-played, the epic fantasy saga that is The Witcher 3: Wild Hunt and its two critically acclaimed expansions, Hearts of Stone and Blood and Wine. Now you’re on the hunt for your next fantasy epic, but which to choose?

Read more...

", "url": "https://kotaku.com/games-like-witcher-3-open-world-rpg-cyberpunk-geralt-1850071540", "title": "15+ Epic Games To Play After The Witcher 3: Wild Hunt", "date_modified": "2023-02-03T19:45:00.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/", "title": "Carving the scheduler out of our orchestrator", "date_modified": "2023-02-02T16:30:28.000Z" }, { "content_html": "Comments", "url": "https://www.hashicorp.com/blog/introducing-hermes-an-open-source-document-management-system", "title": "Hermes, an Open Source Document Management System", "date_modified": "2023-01-31T19:06:31.000Z" }, { "content_html": "Comments", "url": "https://tailscale.dev/blog/weaponizing-hyperfocus", "title": "Weaponizing hyperfocus: Becoming the first DevRel at Tailscale", "date_modified": "2023-01-31T16:17:09.000Z" }, { "content_html": "Comments", "url": "https://tailscale.dev/blog/weaponizing-hyperfocus", "title": "Becoming the first DevRel at Tailscale", "date_modified": "2023-01-31T16:17:09.000Z" }, { "content_html": "Comments", "url": "https://berkeleygraphics.com/public-affairs/bulletins/BT-002/", "title": "Berkeley Mono Ligatures Release", "date_modified": "2023-01-30T17:46:37.000Z" }, { "content_html": "Comments", "url": "https://www.awspuritytest.com/", "title": "AWS Purity Test", "date_modified": "2023-01-27T16:40:30.000Z" }, { "content_html": "Comments", "url": "https://podcast.bitreach.io/episodes/jason-lengstorf", "title": "DevRel should be a process not a project", "date_modified": "2023-01-25T09:25:39.000Z" }, { "content_html": "Comments", "url": "https://containers.dev/", "title": "Development Containers", "date_modified": "2023-01-25T03:03:55.000Z" }, { "content_html": "

Comments

", "url": "https://matt-rickard.com/an-ideal-ci-cd-system", "title": "An Ideal CI/CD System", "date_modified": "2023-01-25T02:07:23.000Z" }, { "content_html": "Comments", "url": "https://astro.build/blog/astro-2/", "title": "Astro 2.0", "date_modified": "2023-01-24T17:48:33.000Z" }, { "content_html": "Comments", "url": "https://github.com/koskimas/kysely", "title": "Kysely: TypeScript SQL Query Builder", "date_modified": "2023-01-24T17:17:06.000Z" }, { "content_html": "Comments", "url": "https://github.com/SigNoz/logs-benchmark", "title": "Elastic, Loki and SigNoz – A Perf Benchmark of Open-Source Logging Platforms", "date_modified": "2023-01-24T08:03:07.000Z" }, { "content_html": "Comments", "url": "https://temporal.io/blog/building-reliable-distributed-systems-in-node", "title": "Building Reliable Distributed Systems in Node.js", "date_modified": "2023-01-21T00:06:43.000Z" }, { "content_html": "Comments", "url": "https://www.mux.com/blog/the-building-blocks-of-great-docs", "title": "The building blocks of great docs", "date_modified": "2023-01-20T22:21:54.000Z" }, { "content_html": "Comments", "url": "https://retool.com/products/mobile", "title": "Show HN: Retool Mobile", "date_modified": "2023-01-19T15:31:21.000Z" }, { "content_html": "Comments", "url": "https://www.mikesukmanowsky.com/blog/authentication-with-django-and-spas", "title": "Use cookies and sessions (not JWTs) for authentication", "date_modified": "2023-01-16T03:24:11.000Z" }, { "content_html": "Comments", "url": "https://github.com/sickcodes/Docker-OSX", "title": "Sickcodes/Docker-OS X: Run macOS VM in a Docker", "date_modified": "2023-01-13T22:20:27.000Z" }, { "content_html": "Comments", "url": "https://sacra.com/p/docker-plg-pivot/", "title": "Docker 2.0 went from $11M to $135M in 2 years", "date_modified": "2023-01-13T18:49:42.000Z" }, { "content_html": "Comments", "url": "https://dagster.io/blog/declarative-scheduling", "title": "Data pipelines are not workflows", "date_modified": "2023-01-09T18:06:01.000Z" }, { "content_html": "Comments", "url": "https://sourcehut.org/blog/2023-01-09-gomodulemirror/", "title": "Sourcehut will blacklist the Go module mirror", "date_modified": "2023-01-09T14:28:39.000Z" }, { "content_html": "
\"Weave
\"Weave

Millions of users rely on Cloudflare WARP to connect to the Internet through Cloudflare’s network. Individuals download the mobile or desktop application and rely on the Wireguard-based tunnel to make their browser faster and more private. Thousands of enterprises trust Cloudflare WARP to connect employees to our Secure Web Gateway and other Zero Trust services as they navigate the Internet.

We’ve heard from both groups of users that they also want to connect to other devices running WARP. Teams can build a private network on Cloudflare’s network today by connecting WARP on one side to a Cloudflare Tunnel, GRE tunnels, or IPSec tunnels on the other end. However, what if both devices already run WARP?

Starting today, we’re excited to make it even easier to build a network on Cloudflare with the launch of WARP-to-WARP connectivity. With a single click, any device running WARP in your organization can reach any other device running WARP. Developers can connect to a teammate's machine to test a web server. Administrators can reach employee devices to troubleshoot issues. The feature works with our existing private network on-ramps, like the tunnel options listed above. All with Zero Trust rules built in.

To get started, sign-up to receive early access to our closed beta. If you’re interested in learning more about how it works and what else we will be launching in the future, keep scrolling.

The bridge to Zero Trust

We understand that adopting a Zero Trust architecture can feel overwhelming at times. With Cloudflare One, our mission is to make Zero Trust prescriptive and approachable regardless of where you are on your journey today. To help users navigate the uncertain, we created resources like our vendor-agnostic Zero Trust Roadmap which lays out a battle-tested path to Zero Trust. Within our own products and services, we’ve launched a number of features to bridge the gap between the networks you manage today and the network you hope to build for your organization in the future.

Ultimately, our goal is to enable you to overlay your network on Cloudflare however you want, whether that be with existing hardware in the field, a carrier you already partner with, through existing technology standards like IPsec tunnels, or more Zero Trust approaches like WARP or Tunnel. It shouldn’t matter which method you chose to start with, the point is that you need the flexibility to get started no matter where you are in this journey. We call these connectivity options on-ramps and off-ramps.

A recap of WARP to Tunnel

The model laid out above allows users to start by defining their specific needs and then customize their deployment by choosing from a set of fully composable on and offramps to connect their users and devices to Cloudflare. This means that customers are able to leverage any of these solutions together to route traffic seamlessly between devices, offices, data centers, cloud environments, and self-hosted or SaaS applications.

One example of a deployment we’ve seen thousands of customers be successful with is what we call WARP-to-Tunnel. In this deployment, the on-ramp Cloudflare WARP ensures end-user traffic reaches Cloudflare’s global network in a secure and performant manner. The off-ramp Cloudflare Tunnel then ensures that, after your Zero Trust rules have been enforced, we have secure, redundant, and reliable paths to land user traffic back in your distributed, private network.

\"Weave

This is a great example of a deployment that is ideal for users that need to support public to private traffic flows (i.e. North-South)

But what happens when you need to support private to private traffic flows (i.e. East-West) within this deployment?

With WARP-to-WARP, connecting just got easier

Starting today, devices on-ramping to Cloudflare with WARP will also be able to off-ramp to each other. With this announcement, we’re adding yet another tool to leverage in new or existing deployments that provides users with stronger network fabric to connect users, devices, and autonomous systems.

\"Weave

This means any of your Zero Trust-enrolled devices will be able to securely connect to any other device on your Cloudflare-defined network, regardless of physical location or network configuration. This unlocks the ability for you to address any device running WARP in the exact same way you are able to send traffic to services behind a Cloudflare Tunnel today. Naturally, all of this traffic flows through our in-line Zero Trust services, regardless of how it gets to Cloudflare, and this new connectivity announced today is no exception.

To power all of this, we now track where WARP devices are connected to, in Cloudflare’s global network, the same way we do for Cloudflare Tunnel. Traffic meant for a specific WARP device is relayed across our network, using Argo Smart Routing, and piped through the transport that routes IP packets to the appropriate WARP device. Since this traffic goes through our Zero Trust Secure Web Gateway — allowing various types of filtering — it means we upgrade and downgrade traffic from purely routed IP packets to fully proxied TLS connections (as well as other protocols). In the case of using SSH to remotely access a colleague’s WARP device, this means that your traffic is eligible for SSH command auditing as well.

Get started today with these use cases

If you already deployed Cloudflare WARP to your organization, then your IT department will be excited to learn they can use this new connectivity to reach out to any device running Cloudflare WARP. Connecting via SSH, RDP, SMB, or any other service running on the device is now simpler than ever. All of this provides Zero Trust access for the IT team members, with their actions being secured in-line, audited, and pushed to your organization’s logs.

Or, maybe you are done with designing a new function of an existing product and want to let your team members check it out at their own convenience. Sending them a link with your private IP — assigned by Cloudflare — will do the job. Their devices will see your machine as if they were in the same physical network, despite being across the other side of the world.

The usefulness doesn’t end with humans on both sides of the interaction: the weekend has arrived, and you have finally set out to move your local NAS to a host provider where you run a virtual machine. By running Cloudflare WARP on it, similarly to your laptop, you can now access your photos using the virtual machine’s private IP. This was already possible with WARP to Tunnel; but with WARP-to-WARP, you also get connectivity in reverse direction, where you can have the virtual machine periodically rsync/scp files from your laptop as well. This means you can make any server initiate traffic towards the rest of your Zero Trust organization with this new type of connectivity.

What’s next?

This feature will be available on all plans at no additional cost. To get started with this new feature, add your name to the closed beta, and we’ll notify you once you’ve been enrolled. Then, you’ll simply ensure that at least two devices are enrolled in Cloudflare Zero Trust and have the latest version of Cloudflare WARP installed.

This new feature builds upon the existing benefits of Cloudflare Zero Trust, which include enhanced connectivity, improved performance, and streamlined access controls. With the ability to connect to any other device in their deployment, Zero Trust users will be able to take advantage of even more robust security and connectivity options.

To get started in minutes, create a Zero Trust account, download the WARP agent, enroll these devices into your Zero Trust organization, and start creating Zero Trust policies to establish fast, secure connectivity between these devices. That’s it.

", "url": "https://blog.cloudflare.com/warp-to-warp/", "title": "Weave your own global, private, virtual Zero Trust network on Cloudflare with WARP-to-WARP", "date_modified": "2023-01-09T14:00:00.000Z" }, { "content_html": "

Comments

", "url": "https://www.sliceofexperiments.com/p/a-comprehensive-guide-for-the-fastest", "title": "A comprehensive guide for the fastest possible Docker builds in human existence", "date_modified": "2023-01-09T03:03:40.000Z" }, { "content_html": "Comments", "url": "https://longform.asmartbear.com/posts/extreme-questions/", "title": "Extreme questions to trigger new, better ideas", "date_modified": "2023-01-09T02:23:39.000Z" }, { "content_html": "Comments", "url": "http://www.mupuf.org/blog/2023/01/04/setting-up-a-ci-system-part-5-time-sharing-your-test-machines/", "title": "Setting Up a CI System Part 5: Time-sharing your test machines", "date_modified": "2023-01-08T03:51:42.000Z" }, { "content_html": "Comments", "url": "https://queue.acm.org/detail.cfm?id=2898444", "title": "Lessons learned from three container-management systems over a decade", "date_modified": "2023-01-07T18:11:41.000Z" }, { "content_html": "Comments", "url": "https://www.enterpriseready.io/", "title": "EnterpriseReady", "date_modified": "2023-01-07T17:22:12.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34287685", "title": "Ask HN: Main things to consider when building an app for business/enterprise", "date_modified": "2023-01-07T13:10:13.000Z" }, { "content_html": "Comments", "url": "https://developers.redhat.com/e-books/podman-action", "title": "Podman in Action", "date_modified": "2023-01-07T05:31:29.000Z" }, { "content_html": "Comments", "url": "https://devbase.fyi/", "title": "Show HN: Devbase – Find products to make your next fantastic project", "date_modified": "2023-01-07T04:24:57.000Z" }, { "content_html": "Comments", "url": "https://github.com/artsy/README", "title": "Artsy Engineering Handbook", "date_modified": "2023-01-06T23:52:13.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34275951", "title": "Ask HN: Why are URNs not more popular?", "date_modified": "2023-01-06T15:44:07.000Z" }, { "content_html": "Comments", "url": "https://clickhouse.com/blog/extracting-converting-querying-local-files-with-sql-clickhouse-local", "title": "Show HN: ClickHouse-local – a small tool for serverless data analytics", "date_modified": "2023-01-05T19:30:48.000Z" }, { "content_html": "Comments", "url": "https://github.com/Owez/yark", "title": "Yark: Advanced and easy YouTube archiver now stable", "date_modified": "2023-01-05T18:45:19.000Z" }, { "content_html": "Comments", "url": "https://www.ycombinator.com/blog/the-yc-founder-directory", "title": "The YC Founder Directory", "date_modified": "2023-01-05T16:00:52.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34255599", "title": "Ask HN: As a startup, what are your must-have sections for a landing page?", "date_modified": "2023-01-05T03:38:52.000Z" }, { "content_html": "Comments", "url": "https://circleci.com/blog/january-4-2023-security-alert/", "title": "Rotate any secrets stored in CircleCI", "date_modified": "2023-01-05T02:57:04.000Z" }, { "content_html": "Comments", "url": "https://circleci.com/blog/january-4-2023-security-alert/", "title": "CircleCI security alert: Rotate any secrets stored in CircleCI", "date_modified": "2023-01-05T02:57:04.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/faster-mysql-with-http3", "title": "Faster MySQL with HTTP/3", "date_modified": "2023-01-04T16:40:26.000Z" }, { "content_html": "Comments", "url": "https://terrateam.io/blog/flying-away-from-aws", "title": "Migrating from AWS to Fly.io", "date_modified": "2023-01-03T21:13:45.000Z" }, { "content_html": "

In the bright afternoon hours of New Year’s Day 2023, I squint through hungover eyes at my phone screen. My Twitter feed is blowing up about some movie I’ve never heard of before: Strange Days, a ‘90s Kathryn Bigelow sci-fi flick starring Ralph Fiennes, Angela Bassett, and Juliette Lewis.

Read more...

", "url": "https://kotaku.com/strange-days-streaming-cyberpunk-2077-braindance-1849945692", "title": "You Should Watch This ‘90s Movie That’s Basically Just Cyberpunk 2077", "date_modified": "2023-01-03T20:55:00.000Z" }, { "content_html": "Comments", "url": "http://blogs.newardassociates.com/blog/2023/you-want-modules-not-microservices.html", "title": "You Want Modules, Not Microservices", "date_modified": "2023-01-03T12:35:03.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34225669", "title": "Ask HN: A Better Docker Compose?", "date_modified": "2023-01-03T00:15:48.000Z" }, { "content_html": "Comments", "url": "https://manual.withcompound.com/chapters/interview-with-benjamin-de-cock-early-designer-at-stripe", "title": "Interview with Benjamin de Cock, early designer at Stripe", "date_modified": "2023-01-01T23:13:11.000Z" }, { "content_html": "Comments", "url": "https://hexmos.com/lama2/index.html", "title": "Show HN: Lama2 - Plain-Text Powered REST API Client for Teams", "date_modified": "2023-01-01T14:06:56.000Z" }, { "content_html": "Comments", "url": "https://workers.tools/", "title": "Worker Tools: Tools for Writing HTTP Servers in Worker Runtimes", "date_modified": "2022-12-31T21:53:51.000Z" }, { "content_html": "Comments", "url": "https://thenewstack.io/product-led-growth-for-dev-first-business-is-it-inevitable/", "title": "Product-led growth for dev-first business: Is it inevitable?", "date_modified": "2022-12-30T22:39:31.000Z" }, { "content_html": "Comments", "url": "https://www.germanvelasco.com/blog/phoenix-1-7-is-view-less", "title": "Phoenix 1.7 is View-less", "date_modified": "2022-12-30T18:52:32.000Z" }, { "content_html": "Comments", "url": "https://erthalion.info/2022/12/30/bpf-performance/", "title": "Running fast and slow: experiments with BPF programs' performance", "date_modified": "2022-12-30T16:23:07.000Z" }, { "content_html": "Comments", "url": "https://prql-lang.org/", "title": "PRQL a simple, powerful, pipelined SQL replacement", "date_modified": "2022-12-30T03:04:44.000Z" }, { "content_html": "Comments", "url": "https://withinboredom.info/blog/2022/12/29/golang-is-evil-on-shitty-networks/", "title": "Golang is evil on shitty networks", "date_modified": "2022-12-29T23:17:43.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/why-we-chose-nanoids-for-planetscales-api", "title": "We Chose NanoIDs for PlanetScale’s API", "date_modified": "2022-12-29T14:38:27.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34163624", "title": "Ask HN: What's a build vs. buy decision that you got wrong?", "date_modified": "2022-12-28T17:49:06.000Z" }, { "content_html": "Comments", "url": "https://dri.es/acquia-cloud-next-a-journey-in-platform-modernization", "title": "Acquia Cloud Next, a journey in platform modernization", "date_modified": "2022-12-26T21:33:52.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=34125628", "title": "Ask HN: What are you predictions for 2023?", "date_modified": "2022-12-25T09:25:01.000Z" }, { "content_html": "Comments", "url": "https://github.com/runfinch/finch", "title": "AWS releases Finch: An open source client for container development", "date_modified": "2022-12-24T08:36:53.000Z" }, { "content_html": "

Henry Cavill may be out as Geralt in the Netflix Witcher series, but we will always have Doug Cockle as Geralt in The Witcher 3: Wild Hunt. And now, thanks to the latest next-gen update to the Witcher 3, we can have the best of both worlds. The new update brings “In The Eternal Fire’s Shadow,” a Witcher-worthy side…

Read more...

", "url": "https://kotaku.com/witcher-3-netflix-armor-quest-next-gen-update-1849920425", "title": "Let’s Get Geralt The Netflix Series Armor In The New Witcher 3 Quest", "date_modified": "2022-12-21T20:25:56.000Z" }, { "content_html": "Comments", "url": "https://www.buildbuddy.io/blog/whats-new-in-bazel-6-0/", "title": "What's New in Bazel 6.0", "date_modified": "2022-12-19T17:44:26.000Z" }, { "content_html": "Comments", "url": "https://www.uber.com/blog/devpod-improving-developer-productivity-at-uber/", "title": "Devpod: Remote development environment at Uber", "date_modified": "2022-12-18T22:44:26.000Z" }, { "content_html": "Comments", "url": "https://queue.acm.org/detail.cfm?id=3570937", "title": "Reinventing backend subsetting at Google", "date_modified": "2022-12-18T07:45:04.000Z" }, { "content_html": "Comments", "url": "https://www.getdbt.com/blog/dbt-cloud-package-update/", "title": "DBT Cloud increase Team plan price by 100% and limit features at the same time", "date_modified": "2022-12-15T17:13:48.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/tailnet-lock/", "title": "Tailnet Lock", "date_modified": "2022-12-14T17:07:07.000Z" }, { "content_html": "Comments", "url": "https://redmonk.com/sogrady/2022/12/09/faster-horse/", "title": "A Faster Horse: Infrastructure was the cloud’s first act. What’s its second?", "date_modified": "2022-12-09T21:06:01.000Z" }, { "content_html": "Comments", "url": "https://neon.tech/blog/serverless-driver-for-postgres/", "title": "Edge-compatible Serverless Driver for Postgres", "date_modified": "2022-12-08T16:18:57.000Z" }, { "content_html": "Comments", "url": "https://github.com/getezy/ezy/releases/tag/v1.0.0-beta.13.2", "title": "Show HN: Ezy – open-source gRPC client, alternative to Postman and Insomnia", "date_modified": "2022-12-08T10:50:05.000Z" }, { "content_html": "Comments", "url": "https://circleci.com/blog/ceo-jim-rose-email-to-circleci-employees/", "title": "CircleCI Layoffs", "date_modified": "2022-12-07T21:23:33.000Z" }, { "content_html": "Comments", "url": "https://git.herrbischoff.com/awesome-macos-command-line/about/", "title": "macOS Command Line", "date_modified": "2022-12-07T16:39:36.000Z" }, { "content_html": "Comments", "url": "https://getconvoy.io/blog/configuring-your-outbound-webhook-requests-with-static-ips/", "title": "Configuring Your Outbound Webhook Requests with Static IPs", "date_modified": "2022-12-07T15:14:02.000Z" }, { "content_html": "Comments", "url": "https://docs.docker.com/desktop/wasm/", "title": "New Docker Desktop: Run WASM Applications Alongside Linux Containers in Docker", "date_modified": "2022-12-07T08:40:00.000Z" }, { "content_html": "Comments", "url": "https://www.nan.fyi/magic-motion", "title": "Framer's Magic Motion", "date_modified": "2022-12-06T01:19:50.000Z" }, { "content_html": "Comments", "url": "https://sifted.eu/articles/vc-scout-programme-problems/", "title": "VCs using scouts means founders get the short end of the stick", "date_modified": "2022-12-05T21:00:38.000Z" }, { "content_html": "Comments", "url": "https://frycos.github.io/vulns4free/2022/12/02/rce-in-20-minutes.html", "title": "Pre-Auth RCE with CodeQL in Under 20 Minutes", "date_modified": "2022-12-03T06:44:34.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2022-12-02-introducing-mona-sans-and-hubot-sans/", "title": "Mona Sans and Hubot Sans", "date_modified": "2022-12-02T17:59:56.000Z" }, { "content_html": "Comments", "url": "https://bugs.chromium.org/p/apvi/issues/detail?id=100", "title": "Android platform signing key compromised", "date_modified": "2022-12-01T22:35:30.000Z" }, { "content_html": "Comments", "url": "https://www.washingtonpost.com/technology/2022/11/30/trustcor-internet-authority-mozilla/", "title": "Mozilla, Microsoft yank TrustCor's root certificate authority", "date_modified": "2022-12-01T01:02:46.000Z" }, { "content_html": "Comments", "url": "https://squeaky.ai/blog/development/how-switching-to-aws-graviton-slashed-our-infrastructure-bill-by-35-percent", "title": "Switching to AWS Graviton slashed our infrastructure bill", "date_modified": "2022-11-30T20:01:31.000Z" }, { "content_html": "

Today I would like to tell you about the next generation of Intel-powered general purpose, compute-optimized, and memory-optimized instances. All three of these instance families are powered by 3rd generation Intel Xeon Scalable processors (Ice Lake) running at 3.5 GHz, and are designed to support your data-intensive workloads with up to 200 Gbps of network bandwidth, the highest EBS performance in EC2 (up to 80 Gbps of bandwidth and up to 350,000 IOPS), and the ability to handle up to twice as many packets per second (PPS) as earlier instances.

\n

New General Purpose (M6in/M6idn) Instances
The original general purpose EC2 instance (m1.small) was launched in 2006 and was the one and only instance type for a little over a year, until we launched the m1.large and m1.xlarge in late 2007. After that, we added the m3 in 2012, m4 in 2015, and the first in a very long line of m5 instances starting in 2017. The family tree branched in 2018 with the addition of the m5d instances with local NVMe storage.

\n

And that brings us to today, and to the new m6in and m6idn instances, both available in 9 sizes:

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
NamevCPUsMemoryLocal Storage
(m6idn only)
Network BandwidthEBS BandwidthEBS IOPS
m6in.large
m6idn.large
28 GiB118 GBUp to 25 GbpsUp to 20 GbpsUp to 87,500
m6in.xlarge
m6idn.xlarge
416 GiB237 GBUp to 30 GbpsUp to 20 GbpsUp to 87,500
m6in.2xlarge
m6idn.2xlarge
832 GiB474 GBUp to 40 GbpsUp to 20 GbpsUp to 87,500
m6in.4xlarge
m6idn.4xlarge
1664 GiB950 GBUp to 50 GbpsUp to 20 GbpsUp to 87,500
m6in.8xlarge
m6idn.8xlarge
32128 GiB1900 GB50 Gbps20 Gbps87,500
m6in.12xlarge
m6idn.12xlarge
48192 GiB2950 GB
(2 x 1425)		75 Gbps30 Gbps131,250
m6in.16xlarge
m6idn.16xlarge
64256 GiB3800 GB
(2 x 1900)		100 Gbps40 Gbps175,000
m6in.24xlarge
m6idn.24xlarge
96384 GiB5700 GB
(4 x 1425)		150 Gbps60 Gbps262,500
m6in.32xlarge
m6idn.32xlarge
128512 GiB7600 GB
(4 x 1900)		200 Gbps80 Gbps350,000
\n

The m6in and m6idn instances are available in the US East (Ohio, N. Virginia) and Europe (Ireland) regions in On-Demand and Spot form. Savings Plans and Reserved Instances are available.

\n

New C6in Instances
Back in 2008 we launched the first in what would prove to be a very long line of Amazon Elastic Compute Cloud (Amazon EC2) instances designed to give you high compute performance and a higher ratio of CPU power to memory than the general purpose instances. Starting with those initial c1 instances, we went on to launch cluster computing instances in 2010 (cc1) and 2011 (cc2), and then (once we got our naming figured out), multiple generations of compute-optimized instances powered by Intel processors: c3 (2013), c4 (2015), and c5 (2016). As our customers put these instances to use in environments where networking performance was starting to become a limiting factor, we introduced c5n instances with 100 Gbps networking in 2018. We also broadened the c5 instance lineup by adding additional sizes (including bare metal), and instances with blazing-fast local NVMe storage.

\n

Today I am happy to announce the latest in our lineup of Intel-powered compute-optimized instances, the c6in, available in 9 sizes:

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
NamevCPUsMemory
Network BandwidthEBS Bandwidth
EBS IOPS
c6in.large24 GiBUp to 25 GbpsUp to 20 GbpsUp to 87,500
c6in.xlarge48 GiBUp to 30 GbpsUp to 20 GbpsUp to 87,500
c6in.2xlarge816 GiBUp to 40 GbpsUp to 20 GbpsUp to 87,500
c6in.4xlarge1632 GiBUp to 50 GbpsUp to 20 GbpsUp to 87,500
c6in.8xlarge3264 GiB50 Gbps20 Gbps87,500
c6in.12xlarge4896 GiB75 Gbps30 Gbps131,250
c6in.16xlarge64128 GiB100 Gbps40 Gbps175,000
c6in.24xlarge96192 GiB150 Gbps60 Gbps262,500
c6in.32xlarge128256 GiB200 Gbps80 Gbps350,000
\n

The c6in instances are available in the US East (Ohio, N. Virginia), US West (Oregon), and Europe (Ireland) Regions.

\n

As I noted earlier, these instances are designed to be able to handle up to twice as many packets per second (PPS) as their predecessors. This allows them to deliver increased performance in situations where they need to handle a large number of small-ish network packets, which will accelerate many applications and use cases includes network virtual appliances (firewalls, virtual routers, load balancers, and appliances that detect and protect against DDoS attacks), telecommunications (Voice over IP (VoIP) and 5G communication), build servers, caches, in-memory databases, and gaming hosts. With more network bandwidth and PPS on tap, heavy-duty analytics applications that retrieve and store massive amounts of data and objects from Amazon Amazon Simple Storage Service (Amazon S3) or data lakes will benefit. For workloads that benefit from low latency local storage, the disk versions of the new instances offer twice as much instance storage versus previous generation.

\n

New Memory-Optimized (R6in/R6idn) Instances
The first memory-optimized instance was the m2, launched in 2009 with the now-quaint Double Extra Large and Quadruple Extra Large names, and a higher ration of memory to CPU power than the earlier m1 instances. We had yet to learn our naming lesson and launched the High Memory Cluster Eight Extra Large (aka cr1.8xlarge) in 2013, before settling on the r prefix and launching r3 instances in 2013, followed by r4 instances in 2014, and r5 instances in 2018.

\n

And again that brings us to today, and to the new r6in and r6idn instances, also available in 9 sizes:

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
NamevCPUsMemoryLocal Storage
(r6idn only)
Network BandwidthEBS BandwidthEBS IOPS
r6in.large
r6idn.large
216 GiB118 GBUp to 25 GbpsUp to 20 GbpsUp to 87,500
r6in.xlarge
r6idn.xlarge
432 GiB237 GBUp to 30 GbpsUp to 20 GbpsUp to 87,500
r6in.2xlarge
r6idn.2xlarge
864 GiB474 GBUp to 40 GbpsUp to 20 GbpsUp to 87,500
r6in.4xlarge
r6idn.4xlarge
16128 GiB950 GBUp to 50 GbpsUp to 20 GbpsUp to 87,500
r6in.8xlarge
r6idn.8xlarge
32256 GiB1900 GB50 Gbps20 Gbps87,500
r6in.12xlarge
r6idn.12xlarge
48384 GiB2950 GB
(2 x 1425)		75 Gbps30 Gbps131,250
r6in.16xlarge
r6idn.16xlarge
64512 GiB3800 GB
(2 x 1900)		100 Gbps40 Gbps175,000
r6in.24xlarge
r6idn.24xlarge
96768 GiB5700 GB
(4 x 1425)		150 Gbps60 Gbps262,500
r6in.32xlarge
r6idn.32xlarge
1281024 GiB7600 GB
(4 x 1900)		200 Gbps80 Gbps350,000
\n

The r6in and r6idn instances are available in the US East (Ohio, N. Virginia), US West (Oregon), and Europe (Ireland) regions in On-Demand and Spot form. Savings Plans and Reserved Instances are available.

\n

Inside the Instances
As you can probably guess from these specs and from the blog post that I wrote to launch the c6in instances, all of these new instance types have a lot in common. I’ll do a rare cut-and-paste from that post in order to reiterate all of the other cool features that are available to you:

\n

Ice Lake Processors – The 3rd generation Intel Xeon Scalable processors run at 3.5 GHz, and (according to Intel) offer a 1.46x average performance gain over the prior generation. All-core Intel Turbo Boost mode is enabled on all instance sizes up to and including the 12xlarge. On the larger sizes, you can control the C-states. Intel Total Memory Encryption (TME) is enabled, protecting instance memory with a single, transient 128-bit key generated at boot time within the processor.

\n

NUMA – Short for Non-Uniform Memory Access, this important architectural feature gives you the power to optimize for workloads where the majority of requests for a particular block of memory come from one of the processors, and that block is “closer” (architecturally speaking) to one of the processors. You can control processor affinity (and take advantage of NUMA) on the 24xlarge and 32xlarge instances.

\n

Networking – Elastic Network Adapter (ENA) is available on all sizes of m6in, m6idn, c6in, r6in, and r6idn instances, and Elastic Fabric Adapter (EFA) is available on the 32xlarge instances. In order to make use of these adapters, you will need to make sure that your AMI includes the latest NVMe and ENA drivers. You can also make use of Cluster Placement Groups.

\n

io2 Block Express – You can use all types of EBS volumes with these instances, including the io2 Block Express volumes that we launched earlier this year. As Channy shared in his post (Amazon EBS io2 Block Express Volumes with Amazon EC2 R5b Instances Are Now Generally Available), these volumes can be as large as 64 TiB, and can deliver up to 256,000 IOPS. As you can see from the tables above, you can use a 24xlarge or 32xlarge instance to achieve this level of performance.

\n

Choosing the Right Instance
Prior to today’s launch, you could choose a c5n, m5n, or r5n instance to get the highest network bandwidth on an EC2 instance, or an r5b instance to have access to the highest EBS IOPS performance and high EBS bandwidth. Now, customers who need high networking or EBS performance can choose from a full portfolio of instances with different memory to vCPU ratio and instance storage options available, by selecting one of c6in, m6in, m6idn, r6in, or r6idn instances.

\n

The higher performance of the c6in instances will allow you to scale your network intensive workloads that need a low memory to vCPU, such as network virtual appliances, caching servers, and gaming hosts.

\n

The higher performance of m6in instances will allow you to scale your network and/or EBS intensive workloads such as data analytics, and telco applications including 5G User Plane Functions (UPF). You have the option to use the m6idn instance for workloads that benefit from low-latency local storage, such as high-performance file systems, or distributed web-scale in-memory caches.

\n

Similarly, the higher network and EBS performance of the r6in instances will allow you to scale your network-intensive SQL, NoSQL, and in-memory database workloads, with the option to use the r6idn when you need low-latency local storage.

\n

\n

— Jeff;

\n

", "url": "https://aws.amazon.com/blogs/aws/new-general-purpose-compute-optimized-and-memory-optimized-amazon-ec2-instances-with-higher-packet-processing-performance/", "title": "New General Purpose, Compute Optimized, and Memory-Optimized Amazon EC2 Instances with Higher Packet-Processing Performance", "date_modified": "2022-11-29T03:57:07.000Z" }, { "content_html": "Comments", "url": "https://img.ly/blog/ultimate-guide-to-ffmpeg/", "title": "The Ultimate Guide to FFmpeg", "date_modified": "2022-11-28T09:30:20.000Z" }, { "content_html": "

AWS VP and Chief Evangelist Jeff Barr, plus a select group of AWS Developer Advocate colleagues, have personally chosen their picks for some of the most impactful and exciting product and service launches to debut at AWS re:Invent 2022. From now through Dec. 1, we’ll update this page daily with links to their AWS News Blog posts (plus a few noteworthy preview posts) so you can dive deeper once the launches have been announced.

\n

As always, there’s simply too much for the team to cover and even if a launch doesn’t make this list, that doesn’t mean it’s not noteworthy. Make sure to check out What’s New for a complete rundown of all the AWS re:Invent 2022 announcements.

\n

Here are a few more resources to help you keep up with all the re:Invent news:

\n \n
\n(This post was updated: 8:31 p.m. PST, Nov. 28, 2022.)\n

\n
\n

\n
\nQuick category links:\n

\n

Analytics | Business Applications | Artificial Intelligence / Machine Learning | Compute | Containers | Database | Management Tools | Migration & Transfer Services | Security, Identity, & Compliance | Storage |

\n
\n

Analytics

\n

New — Create and Share Operational Reports at Scale with Amazon QuickSight Paginated Reports
This feature allows customers to create and share highly formatted, personalized reports containing business-critical data to hundreds of thousands of end-users without any infrastructure setup or maintenance, up-front licensing, or long-term commitments.

\n

New Amazon QuickSight API Capabilities to Accelerate Your BI Transformation
New QuickSight API capabilities allow programmatic creation and management of dashboards, analysis, and templates.

\n

New AWS Glue 4.0 – New and Updated Engines, More Data Formats, and More
This version of Glue includes Python 3.10 and Apache Spark 3.3.0, plus native support for the Cloud Shuffle Service Plugin for Spark. It also includes Pandas support, and more.

\n

Announcing AWS Glue for Ray (Preview)
Data engineers can use AWS Glue for Ray to process large datasets with Python and popular Python libraries.

\n

New for Amazon Transcribe – Real-Time Analytics During Live Calls
Real-time call analytics provides APIs for developers to accurately transcribe live calls and at the same time identify customer experience issues and sentiment in real time.

\n
\n

Artificial Intelligence / Machine Learning

\n

Classifying and Extracting Mortgage Loan Data with Amazon Textract
The new API was created in response to requests from major lenders in the industry to help them process applications faster and reduce errors, which improves the end-customer experience and lowers operating costs.

\n

Amazon CodeWhisperer Adds Enterprise Administrative Controls, Simple Sign-up, and Support for New Languages (Preview)
Administrators can now easily integrate CodeWhisperer with their existing workforce identity solutions, provide access to users and groups, and configure organization-wide settings.

\n
\n

Business Applications

\n

AWS Wickr – A Secure, End-to-End Encrypted Communication Service For Enterprises With Auditing And Regulatory Requirements
Unlike many enterprise communication tools, Wickr uses end-to-end encryption mechanisms to ensure your messages, files, voice, or video calls are solely accessible to their intended recipients.

\n
\n

Compute

\n

New – ENA Express: Improved Network Latency and Per-Flow Performance on EC2
Jeff Barr shares how ENA Express gives you a lot more per-flow bandwidth with a lot less variability.

\n

New General Purpose, Compute Optimized, and Memory-Optimized Amazon EC2 Instances with Higher Packet-Processing Performance
The new instance families are designed to support your data-intensive workloads with the highest EBS performance in EC2, and the ability to handle up to twice as many packets per second (PPS) as earlier instances.

\n

New Amazon EC2 Instance Types In the Works – C7gn, R7iz, and Hpc7g
Jeff Barr provides a look at three upcoming and exciting new instance types.

\n

New – Amazon ECS Service Connect Enables Easy Communication Between Microservices
This new capability simplifies building and operating resilient distributed applications. You can add a layer of resilience to your ECS service communication and get traffic insights with no changes to your application code.

\n

Announcing the availability of Microsoft Office Amazon Machine Images (AMIs) on Amazon EC2 with AWS provided licenses
With this offering, customers have the flexibility to run Microsoft Office dependent applications on EC2.

\n
\n

Containers

\n

New – AWS Marketplace for Containers Now Supports Direct Deployment to Amazon EKS Clusters
This new launch makes it easier for you to find third-party Kubernetes operation software from the Amazon EKS console and deploy it to your EKS clusters using the same commands used to deploy EKS add-ons.

\n
\n

Database

\n

New – Amazon RDS Optimized Reads and Optimized Writes
These two new features will accelerate your Amazon RDS for MySQL workloads.

\n

New – Fully Managed Blue/Green Deployments in Amazon Aurora and Amazon RDS
This new feature for Amazon Aurora with MySQL compatibility, Amazon RDS for MySQL, and Amazon RDS for MariaDB, enables you to make database updates safer, simpler, and faster.

\n
\n

Management Tools

\n

New – AWS Config Rules Now Support Proactive Compliance
This release extends AWS Config rules to support proactive mode so that they can be run at any time before provisioning and save time spent to implement custom pre-deployment validations.

\n

New for AWS Control Tower – Comprehensive Controls Management (Preview)
You can use the new capability to apply managed preventative, detective, and proactive controls to accounts and organizational units by service, control objective, or compliance framework.

\n

Protect Sensitive Data with Amazon CloudWatch Logs
This new set of capabilities for Amazon CloudWatch Logs leverages pattern matching and machine learning (ML) to detect and protect sensitive log data in transit.

\n

New – Amazon CloudWatch Cross-Account Observability
This new capability lets you search, analyze, and correlate cross-account telemetry data stored in CloudWatch such as metrics, logs, and traces.

\n
\nAmazon CloudWatch Internet Monitor Provides End-to-End Visibility into Internet Performance for your Applications (Preview)\n
This new capability gives visibility into how an internet issue might impact the performance and availability of your applications. It allows you to reduce the time it takes to diagnose internet issues from days to minutes.\n

\n

Migration & Transfer Services

\n

New – A Fully Managed Schema Conversion in AWS Database Migration Service
AWS DMS Schema Conversion streamlines database migrations by making schema assessment and conversion available inside AWS DMS. You can now plan, assess, convert and migrate under one central DMS service.

\n

AWS Application Migration Service Major Updates – New Migration Servers Grouping, Updated Launch, and Post-Launch Template
These three major updates will support your migration projects of any size.

\n
\n

Security, Identity & Compliance

\n

Amazon Inspector Now Scans AWS Lambda Functions for Vulnerabilities
Until now, customers who wanted to analyze their mixed workloads (including EC2 instances, container images, and Lambda functions) against common vulnerabilities needed to use AWS and third-party tools.

\n

Automated Data Discovery for Amazon Macie
This new capability allows you to gain visibility into where your sensitive data resides on Amazon Simple Storage Service (Amazon S3) at a fraction of the cost of running a full data inspection across all your S3 buckets.

\n

AWS announces Amazon Verified Permissions (Preview)
This central fine-grained permissions management system simplifies changing and updating permission rules in a single place without needing to change the code.

\n

Storage

\n

New – Failover Controls for Amazon S3 Multi-Region Access Points
These controls let you shift S3 data access request traffic routed through an Amazon S3 Multi-Region Access Point to an alternate AWS Region within minutes to test and build highly available applications for business continuity.

\n

New – Announcing Amazon EFS Elastic Throughput
This new throughput mode is designed to provide your applications with as much throughput as they need with pay-as-you-use pricing.

\n

New for AWS Backup – Protect and Restore Your CloudFormation Stacks
You now have an automated solution to create and restore your applications with a simplified experience, eliminating the need to manage custom scripts.

\n

New – Amazon Redshift Support in AWS Backup
AWS Backup allows you to define a central backup policy to manage data protection of your applications and can now also protect your Amazon Redshift clusters.

\n

Announcing Automated in-AWS Failback for AWS Elastic Disaster Recovery
The new automated support provides a simplified and expedited experience to fail back Amazon Elastic Compute Cloud (Amazon EC2) instances to the original Region, and both failover and failback processes (for on-premises or in-AWS recovery) can be conveniently started from the AWS Management Console.

", "url": "https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2022/", "title": "Top Announcements of AWS re:Invent 2022", "date_modified": "2022-11-28T06:08:04.000Z" }, { "content_html": "Comments", "url": "https://cyrilgrislain.substack.com/p/startup-restructuring-101", "title": "Startup Restructuring 101", "date_modified": "2022-11-27T01:00:07.000Z" }, { "content_html": "Comments", "url": "https://github.com/nektos/act", "title": "Act: Run your GitHub Actions locally", "date_modified": "2022-11-26T07:21:31.000Z" }, { "content_html": "Comments", "url": "https://aws.amazon.com/blogs/opensource/introducing-finch-an-open-source-client-for-container-development/", "title": "Finch: An open-source client for container development", "date_modified": "2022-11-25T19:16:46.000Z" }, { "content_html": "Comments", "url": "https://paulstamatiou.com/craft/", "title": "Craft", "date_modified": "2022-11-25T03:31:15.000Z" }, { "content_html": "Comments", "url": "https://github.com/getlago/lago/wiki/Per-Seat-pricing-is-off-the-table-now", "title": "Per-seat pricing is off the table now", "date_modified": "2022-11-23T17:15:32.000Z" }, { "content_html": "Comments", "url": "https://wasmer.io/posts/announcing-wasmer-3.0", "title": "Wasmer 3.0", "date_modified": "2022-11-23T17:01:03.000Z" }, { "content_html": "Comments", "url": "https://emily.id.au/tailscale", "title": "CVE-2022-41924 – tailscaled can be used to remotely execute code on Windows", "date_modified": "2022-11-21T18:17:54.000Z" }, { "content_html": "Comments", "url": "https://www.kite.com/blog/product/kite-is-saying-farewell/", "title": "Kite is saying farewell, and is open-sourcing all of its code.", "date_modified": "2022-11-20T20:57:33.000Z" }, { "content_html": "Comments", "url": "https://www.kite.com/blog/product/kite-is-saying-farewell/", "title": "Kite is saying farewell and open-sourcing its code", "date_modified": "2022-11-20T20:57:33.000Z" }, { "content_html": "Comments", "url": "https://www.heavybit.com/library/podcasts/the-kubelist-podcast/ep-33-tailscale-with-avery-pennarun", "title": "Tailscale with Avery Pennarun", "date_modified": "2022-11-20T19:23:02.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=33682599", "title": "Ask HN: What is example of good documentation in your opinion?", "date_modified": "2022-11-20T16:50:34.000Z" }, { "content_html": "Comments", "url": "http://bradconte.com/files/misc/HackerNewsParodyThread/", "title": "Hacker News Parody Thread", "date_modified": "2022-11-20T13:14:22.000Z" }, { "content_html": "Comments", "url": "https://jay.gooby.org/2022/05/06/use-a-basic-gmail-account-to-send-mail-as-with-a-domain-that-uses-cloudflare-email-routing", "title": "Use a custom domain to send emails with Gmail using Cloudflare email routing", "date_modified": "2022-11-18T19:02:20.000Z" }, { "content_html": "Comments", "url": "https://blog.alexellis.io/blazing-fast-ci-with-microvms/", "title": "Blazing fast CI with MicroVMs", "date_modified": "2022-11-18T16:09:33.000Z" }, { "content_html": "Comments", "url": "https://blog.alexellis.io/blazing-fast-ci-with-microvms/", "title": "Fast CI with MicroVMs", "date_modified": "2022-11-18T16:09:33.000Z" }, { "content_html": "Comments", "url": "https://devenv.sh/", "title": "Devenv.sh: Fast and reproducible developer environments using Nix", "date_modified": "2022-11-18T14:52:34.000Z" }, { "content_html": "
\"Doubling
\"Doubling

Local development gives you a fully-controllable and easy-to-debug testing environment. At the start of this year, we brought this experience to Workers developers by launching Miniflare 2.0: a local Cloudflare Workers simulator. Miniflare 2 came with features like step-through debugging support, detailed console.logs, pretty source-mapped error pages, live reload and a highly-configurable unit testing environment. Not only that, but we also incorporated Miniflare into Wrangler, our Workers CLI, to enable wrangler dev’s --local mode.

Today, we’re taking local development to the next level! In addition to introducing new support for migrating existing projects to your local development environment, we're making it easier to work with your remote data—locally! Most importantly, we're releasing a much more accurate Miniflare 3, powered by the recently open-sourced workerd runtime—the same runtime used by Cloudflare Workers!

Enabling local development with workerd

One of the superpowers of having a local development environment is that you can test changes without affecting users in production. A great local environment offers a level of fidelity on par with production.

The way we originally approached local development was with Miniflare 2, which reimplemented Workers runtime APIs in JavaScript. Unfortunately, there were subtle behavior mismatches between these re-implementations and the real Workers runtime. These types of issues are really difficult for developers to debug, as they don’t appear locally, and step-through debugging of deployed Workers isn’t possible yet. For example, the following Worker returns responses successfully in Miniflare 2, so we might assume it’s safe to publish:

let cachedResponsePromise;\nexport default {\n  async fetch(request, env, ctx) {\n    // Let's imagine this fetch takes a few seconds. To speed up our worker, we\n    // decide to only fetch on the first request, and reuse the result later.\n    // This works fine in Miniflare 2, so we must be good right?\n    cachedResponsePromise ??= fetch(\"https://example.com\");\n    return (await cachedResponsePromise).clone();\n  },\n};\n
\n

However, as soon as we send multiple requests to our deployed Worker, it fails with Error: Cannot perform I/O on behalf of a different request. The problem here is that response bodies created in one request’s handler cannot be accessed from a different request's handler. This limitation allows Cloudflare to improve overall Worker performance, but it was almost impossible for Miniflare 2 to detect these types of issues locally. In this particular case, the best solution is to cache using fetch itself.

Additionally, because the Workers runtime uses a very recent version of V8, it supports some JavaScript features that aren’t available in all versions of Node.js. This meant a few features implemented in Workers, like Array#findLast, weren’t always available in Miniflare 2.

With the Workers runtime now open-sourced, Miniflare 3 can leverage the same implementations that are deployed on Cloudflare’s network, giving bug-for-bug compatibility and practically eliminating behavior mismatches. 🎉

\"Doubling
Miniflare 3’s new simplified architecture using worked

This radically simplifies our implementation too. We were able to remove over 50,000 lines of code from Miniflare 2. Of course, we still kept all the Miniflare special-sauce that makes development fun like live reload and detailed logging. 🙂

\"Doubling\n

Local development with real data

We know that many developers choose to test their Workers remotely on the Cloudflare network as it gives them the ability to test against real data. Testing against fake data in staging and local environments is sometimes difficult, as it never quite matches the real thing.

With Miniflare 3, we’re blurring the lines between local and remote development, by bringing real data to your machine as an experimental opt-in feature. If enabled, Miniflare will read and write data to namespaces on the Cloudflare network, as your Worker would when deployed. This is only supported with Workers KV for now, but we’re exploring similar solutions for R2 and D1.

\"Doubling
Miniflare’s system for accessing real KV data, reads and writes are cached locally for future accesses

A new default for Wrangler

With Miniflare 3 now effectively as accurate as the real Workers environment, and the ability to access real data locally, we’re revisiting the decision to make remote development the initial Wrangler experience. In a future update, wrangler dev --local will become the default. --local will no longer be required. Benchmarking suggests this will bring an approximate 10x reduction to startup and a massive 60x reduction to script reload times! Over the next few weeks, we’ll be focusing on further optimizing Wrangler’s performance to bring you the fastest Workers development experience yet!

\"Doubling

wrangler init --from-dash

We want all developers to be able to take advantage of the improved local experience, so we’re making it easy to start a local Wrangler project from an existing Worker that’s been developed in the Cloudflare dashboard. With Node.js installed, run npx wrangler init --from-dash <your_worker_name>in your terminal to set up a new project with all your existing code and bindings such as KV namespaces configured. You can now seamlessly continue development of your application locally, taking advantage of all the developer experience improvements Wrangler and Miniflare provide. When you’re ready to deploy your worker, run npx wrangler publish.

Looking to the future

Over the next few months, the Workers team is planning to further improve the local development experience with a specific focus on automated testing. Already, we’ve released a preliminary API for programmatic end-to-end tests with wrangler dev, but we’re also investigating ways of bringing Miniflare 2’s Jest/Vitest environments to workerd. We’re also considering creating extensions for popular IDEs to make developing workers even easier. 👀

Miniflare 3.0 is now included in Wrangler! Try it out by running npx wrangler@latest dev --experimental-local. Let us know what you think in the #wrangler channel on the Cloudflare Developers Discord, and please open a GitHub issue if you hit any unexpected behavior.

", "url": "https://blog.cloudflare.com/miniflare-and-workerd/", "title": "Doubling down on local development with Workers: Miniflare meets workerd", "date_modified": "2022-11-18T14:00:00.000Z" }, { "content_html": "Comments", "url": "https://lassebomh.github.io/box-breathing/", "title": "Show HN: A visual guide to Box Breathing", "date_modified": "2022-11-18T12:16:26.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/introducing-tailscale-funnel/", "title": "Tailscale Funnel", "date_modified": "2022-11-18T00:57:27.000Z" }, { "content_html": "Comments", "url": "https://github.com/wbkd/awesome-node-based-uis", "title": "Awesome Node-Based UIs", "date_modified": "2022-11-17T14:51:01.000Z" }, { "content_html": "
\"The
\"The

Today, we are announcing the general availability of OpenAPI Schemas for the Cloudflare API. These are published via GitHub and will be updated regularly as Cloudflare adds and updates APIs. OpenAPI is the widely adopted standard for defining APIs in a machine-readable format. OpenAPI Schemas allow for the ability to plug our API into a wide breadth of tooling to accelerate development for ourselves and customers. Internally, it will make it easier for us to maintain and update our APIs. Before getting into those benefits, let’s start with the basics.

What is OpenAPI?

Much of the Internet is built upon APIs (Application Programming Interfaces) or provides them as services to clients all around the world. This allows computers to talk to each other in a standardized fashion. OpenAPI is a widely adopted standard for how to define APIs. This allows other machines to reliably parse those definitions and use them in interesting ways. Cloudflare’s own API Shield product uses OpenAPI schemas to provide schema validation to ensure only well-formed API requests are sent to your origin.

Cloudflare itself has an API that customers can use to interface with our security and performance products from other places on the Internet. How do we define our own APIs? In the past we used a standard called JSON Hyper-Schema. That had served us well, but as time went on we wanted to adopt more tooling that could both benefit ourselves internally and make our customer’s lives easier. The OpenAPI community has flourished over the past few years providing many capabilities as we will discuss that were unavailable while we used JSON Hyper-Schema. As of today we now use OpenAPI.

You can learn more about OpenAPI itself here. Having an open, well-understood standard for defining our APIs allows for shared tooling and infrastructure to be used that can read these standard definitions. Let’s take a look at a few examples.

Uses of Cloudflare’s OpenAPI schemas

Most customers won’t need to use the schemas themselves to see value. The first system leveraging OpenAPI schemas is our new API Docs that were announced today. Because we now have OpenAPI schemas, we leverage the open source tool Stoplight Elements to aid in generating this new doc site. This allowed us to retire our previously custom-built site that was hard to maintain. Additionally, many engineers at Cloudflare are familiar with OpenAPI, so we gain teams can write new schemas more quickly and are less likely to make mistakes by using a standard that teams understand when defining new APIs.

There are ways to leverage the schemas directly, however. The OpenAPI community has a huge number of tools that only require a set of schemas to be able to use. Two such examples are mocking APIs and library generation.

Mocking Cloudflare’s API

Say you have code that calls Cloudflare’s API and you want to be able to easily run unit tests locally or integration tests in your CI/CD pipeline. While you could just call Cloudflare’s API in each run, you may not want to for a few reasons. First, you may want to run tests frequently enough that managing the creation and tear down of resources becomes a pain. Also, in many of these tests you aren’t trying to validate logic in Cloudflare necessarily, but your own system’s behavior. In this case, mocking Cloudflare’s API would be ideal since you can gain confidence that you aren’t violating Cloudflare’s API contract, but without needing to worry about specifics of managing real resources. Additionally, mocking allows you to simulate different scenarios, like being rate limited or receiving 500 errors. This allows you to test your code for typically rare circumstances that can end up having a serious impact.

As an example, Spotlight Prism could be used to mock Cloudflare’s API for testing purposes. With a local copy of Cloudflare’s API Schemas you can run the following command to spin up a local mock server:

$ docker run --init --rm \\\n  -v /home/user/git/api-schemas/openapi.yaml:/tmp/openapi.yaml \\\n  -p 4010:4010 stoplight/prism:4 \\\n  mock -h 0.0.0.0 /tmp/openapi.yaml\n
\n

Then you can send requests to the mock server in order to validate that your use of Cloudflare’s API doesn’t violate the API contract locally:

$ curl -sX PUT localhost:4010/zones/f00/activation_check \\\n  -Hx-auth-email:foo@bar.com -Hx-auth-key:foobarbaz | jq\n{\n  \"success\": true,\n  \"errors\": [],\n  \"messages\": [],\n  \"result\": {\n    \"id\": \"023e105f4ecef8ad9ca31a8372d0c353\"\n  }\n}\n
\n

This means faster development and shorter test runs while still catching API contract issues early before they get merged or deployed.

Library generation

Cloudflare has libraries in many programming languages like Terraform and Go, but we don’t support every possible programming language. Fortunately, using a tool like openapi generator, you can feed in Cloudflare’s API schemas and generate a library in a wide range of languages to then use in your code to talk to Cloudflare’s API. For example, you could generate a Java library using the following commands:

git clone https://github.com/openapitools/openapi-generator\ncd openapi-generator\nmvn clean package\njava -jar modules/openapi-generator-cli/target/openapi-generator-cli.jar generate \\\n   -i https://raw.githubusercontent.com/cloudflare/api-schemas/main/openapi.yaml \\\n   -g java \\\n   -o /var/tmp/java_api_client\n
\n

And then start using that client in your Java code to talk to Cloudflare’s API.

How Cloudflare transitioned to OpenAPI

As mentioned earlier, we previously used JSON Hyper-Schema to define our APIs. We have roughly 600 endpoints that were already defined in the schemas. Here is a snippet of what one endpoint looks like in JSON Hyper-Schema:

{\n      \"title\": \"List Zones\",\n      \"description\": \"List, search, sort, and filter your zones.\",\n      \"rel\": \"collection\",\n      \"href\": \"zones\",\n      \"method\": \"GET\",\n      \"schema\": {\n        \"$ref\": \"definitions/zone.json#/definitions/collection_query\"\n      },\n      \"targetSchema\": {\n        \"$ref\": \"#/definitions/response_collection\"\n      },\n      \"cfOwnership\": \"www\",\n      \"cfPlanAvailability\": {\n        \"free\": true,\n        \"pro\": true,\n        \"business\": true,\n        \"enterprise\": true\n      },\n      \"cfPermissionsRequired\": {\n        \"enum\": [\n          \"#zone:read\"\n        ]\n      }\n    }\n
\n

Let’s look at the same endpoint in OpenAPI:

/zones:\n    get:\n      description: List, search, sort, and filter your zones.\n      operationId: zone-list-zones\n      responses:\n        4xx:\n          content:\n            application/json:\n              schema:\n                allOf:\n                - $ref: '#/components/schemas/components-schemas-response_collection'\n                - $ref: '#/components/schemas/api-response-common-failure'\n          description: List Zones response failure\n        \"200\":\n          content:\n            application/json:\n              schema:\n                $ref: '#/components/schemas/components-schemas-response_collection'\n          description: List Zones response\n      security:\n      - api_email: []\n        api_key: []\n      summary: List Zones\n      tags:\n      - Zone\n      x-cfPermissionsRequired:\n        enum:\n        - '#zone:read'\n      x-cfPlanAvailability:\n        business: true\n        enterprise: true\n        free: true\n        pro: true\n
\n

You can see that the two look fairly similar and for the most part the same information is contained in each including method type, a description, and request and response definitions (although those are linked in $refs). The value of migrating from one to the other isn’t the change in how we define the schemas themselves, but in what we can do with these schemas. Numerous tools can parse the latter, the OpenAPI, while much fewer can parse the former, the JSON Hyper-Schema.

If this one API was all that made up the Cloudflare API, it would be easy to just convert the JSON Hyper-Schema into the OpenAPI Schema by hand and call it a day. Doing this 600 times, however, was going to be a huge undertaking. When considering that teams are constantly adding new endpoints, it would be impossible to keep up. It was also the case that our existing API docs used the existing JSON Hyper-Schema, so that meant that we would need to keep both schemas up to date during any transition period. There had to be a better way.

Auto conversion

Given both JSON Hyper-Schema and OpenAPI are standards, it reasons that it should be possible to take a file in one format and convert to the other, right? Luckily the answer is yes! We built a tool that took all existing JSON Hyper-Schema and output fully compliant OpenAPI schemas. This of course didn’t happen overnight, but because of existing OpenAPI tooling, we could iteratively improve the auto convertor and run OpenAPI validation tooling over the output schemas to see what issues the conversion tool still had.

After many iterations and improvements to the conversion tool, we finally had fully compliant OpenAPI Spec schemas being auto-generated from our existing JSON Hyper-Schema. While we were building this tool, teams kept adding and updating the existing schemas and our Product Content team was also updating text in the schemas to make our API docs easier to use. The benefit of this process is we didn’t have to slow any of that work down since anything that changed in the old schemas was automatically reflected in the new schemas!

Once the tool was ready, the remaining step was to decide when and how we would stop making updates to the JSON Hyper-Schemas and move all teams to the OpenAPI Schemas. The (now old) API docs were the biggest concern, given they only understood JSON Hyper-Schema. Thanks to the help of our Developer Experience and Product Content teams, we were able to launch the new API docs today and can officially cut over to OpenAPI today as well!

What’s next?

Now that we have fully moved over to OpenAPI, more opportunities become available. Internally, we will be investigating what tooling we can adopt in order to help reduce the effort of individual teams and speed up API development. One idea we are exploring is automatically creating openAPI schemas from code notations. Externally, we now have the foundational tools necessary to begin exploring how to auto generate and support more programming language libraries for customers to use. We are also excited to see what you may do with the schemas yourself, so if you do something cool or have ideas, don’t hesitate to share them with us!

", "url": "https://blog.cloudflare.com/open-api-transition/", "title": "The Cloudflare API now uses OpenAPI schemas", "date_modified": "2022-11-16T14:00:00.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/how-planetscale-boost-serves-your-sql-queries-instantly", "title": "How PlanetScale Boost serves SQL queries faster", "date_modified": "2022-11-15T16:58:54.000Z" }, { "content_html": "Comments", "url": "https://bob-cd.github.io/", "title": "Bob CD", "date_modified": "2022-11-14T09:38:25.000Z" }, { "content_html": "

Comments

", "url": "https://flightaware.engineering/taking-off-with-nix-at-flightaware/", "title": "Taking off with Nix at FlightAware", "date_modified": "2022-11-14T01:09:36.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=33582687", "title": "Ask HN: What are your “scratch own itch” projects?", "date_modified": "2022-11-13T13:07:21.000Z" }, { "content_html": "Comments", "url": "https://pickard.cc/posts/why-does-zsh-start-slowly/", "title": "Why does zsh start so slowly?", "date_modified": "2022-11-13T05:28:30.000Z" }, { "content_html": "Comments", "url": "https://infosec.rodeo/posts/thoughts-on-aws-iam/", "title": "AWS IAM Roles, a tale of unnecessary complexity", "date_modified": "2022-11-11T20:34:16.000Z" }, { "content_html": "Comments", "url": "https://github.com/mona-sans", "title": "GitHub is releasing two open-source fonts: Mona and Hubot Sans", "date_modified": "2022-11-10T21:28:38.000Z" }, { "content_html": "Comments", "url": "https://podman-desktop.io", "title": "Podman Desktop: A Free OSS Alternative to Docker Desktop", "date_modified": "2022-11-09T19:55:40.000Z" }, { "content_html": "Comments", "url": "https://affinity.serif.com/en-gb/", "title": "Affinity 2", "date_modified": "2022-11-09T12:05:32.000Z" }, { "content_html": "Comments", "url": "https://earthly.dev/blog/chroot/", "title": "Let’s build a container runtime using only the chroot system call", "date_modified": "2022-11-09T01:42:04.000Z" }, { "content_html": "Comments", "url": "https://blessed.rs/crates", "title": "Blessed.rs – An unofficial guide to the Rust ecosystem", "date_modified": "2022-11-07T14:25:42.000Z" }, { "content_html": "

Comments

", "url": "https://stripe.com/blog/fast-secure-builds-choose-two", "title": "Fast builds, secure builds. Choose two", "date_modified": "2022-11-07T09:49:28.000Z" }, { "content_html": "Comments", "url": "https://www.principlesofpricing.com/", "title": "The Principles of Pricing", "date_modified": "2022-11-07T07:54:09.000Z" }, { "content_html": "Comments", "url": "https://www.principlesofpricing.com/", "title": "Principles of Pricing", "date_modified": "2022-11-07T07:54:09.000Z" }, { "content_html": "Comments", "url": "https://blakewatson.com/journal/almost-monospaced-the-perfect-fonts-for-writing/", "title": "Almost monospaced: the perfect fonts for writing", "date_modified": "2022-11-06T23:31:28.000Z" }, { "content_html": "Comments", "url": "https://ebpf.io/what-is-ebpf/", "title": "eBPF – Adding functionality to OS at runtime to achieve performance and security", "date_modified": "2022-11-06T06:55:56.000Z" }, { "content_html": "Comments", "url": "https://sysdig.com/blog/kernel-parameters-falco/", "title": "Tales from the Kernel Parameter Side", "date_modified": "2022-11-04T07:52:16.000Z" }, { "content_html": "Comments", "url": "https://github.com/apenwarr/blip", "title": "Blip: A tool for seeing your Internet latency", "date_modified": "2022-11-03T00:52:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/tierrun", "title": "Show HN: Tier.run – Terraform for Stripe", "date_modified": "2022-11-02T00:40:29.000Z" }, { "content_html": "Comments", "url": "https://www.rewind.ai", "title": "Rewind: The Search Engine for Your Life", "date_modified": "2022-11-01T14:34:08.000Z" }, { "content_html": "Comments", "url": "https://ismypackagereproducibleyet.org/", "title": "Is My Package Reproducible Yet?", "date_modified": "2022-11-01T04:25:57.000Z" }, { "content_html": "

Duel Corp. looks so pretty that you’d be forgiving for thinking that it’s an upcoming RPG from Square Enix. But it’s actually made by some indie developers who decided to make a Soulslike from retro pixel art. And the effect is incredible.

Read more...

", "url": "https://kotaku.com/duel-corp-soulslike-rpg-octopath-traveler-dark-souls-1849724755", "title": "Gorgeous Retro Soulslike RPG Looks Just Like Octopath Traveler", "date_modified": "2022-10-31T20:00:59.000Z" }, { "content_html": "Comments", "url": "https://ffmpeg.guide", "title": "Show HN: FFmpeg Command Visualizer and Editor", "date_modified": "2022-10-29T11:20:02.000Z" }, { "content_html": "Comments", "url": "https://buf.build/blog/protobuf-es-the-protocol-buffers-typescript-javascript-runtime-we-all-deserve", "title": "Protobuf-ES – Implementation of Protocol Buffers for TypeScript and JavaScript", "date_modified": "2022-10-29T06:34:59.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/ssh-console/", "title": "Making an SSH client the hard way", "date_modified": "2022-10-27T17:13:44.000Z" }, { "content_html": "Comments", "url": "https://github.com/charmbracelet/vhs", "title": "VHS: Your CLI home video recorder", "date_modified": "2022-10-27T14:27:09.000Z" }, { "content_html": "Comments", "url": "https://github.com/charmbracelet/vhs", "title": "VHS: CLI home video recorder", "date_modified": "2022-10-27T14:27:09.000Z" }, { "content_html": "Comments", "url": "https://www.influxdata.com/blog/influxdb-engine/", "title": "IOx: InfluxData’s New Storage Engine", "date_modified": "2022-10-26T14:51:58.000Z" }, { "content_html": "Comments", "url": "https://www.cloudflarestatus.com/incidents/kdpqngcbbn25", "title": "Cloudflare CDN Partial Outage", "date_modified": "2022-10-25T17:07:36.000Z" }, { "content_html": "Comments", "url": "https://taras.glek.net/post/curious-case-of-maintaining-sufficient-free-space-with-zfs/", "title": "Curious Case of Maintaining Sufficient Free Space with ZFS", "date_modified": "2022-10-23T20:59:16.000Z" }, { "content_html": "Comments", "url": "https://taras.glek.net/post/curious-case-of-maintaining-sufficient-free-space-with-zfs/", "title": "Maintaining sufficient free space with ZFS", "date_modified": "2022-10-23T20:59:16.000Z" }, { "content_html": "

gRPC is a modern open source high performance Remote Procedure Call (RPC) framework that can run in any environment. It plays a critical role in efficiently connecting microservices in and across data centers with pluggable support for load balancing, tracing, health checking, authentication and other cross-cutting features. It may also be applied in the last mile of distributed computing to connect devices, mobile applications and browsers to backend services hosted on the public cloud. This unique position in the software stack can provide a clear end-to-end view of the whole system. A new gRPC observability feature provides this clarity for workloads running on, and/or able to connect to, Google Cloud.

The kinds of observability data provided

gRPC observability provides three different types of data:

1. Logs for key RPC events, including:

2. Metrics (or statistical data) for key RPC events, including:

3. Distributed traces for RPCs and their fanout RPCs across the system. For example, when serving an RPC from upstream, a server may need to create multiple RPCs to its own backends. The distributed trace helps the user understand the relationships between these RPCs, the latency for each of them, and key events happening throughout the system.

How the observability data is produced and collected

When developers enable the gRPC observability feature in their binaries, the gRPC library will report the logging, metrics, and tracing data to Google Cloud’s operations suite. Once the observability data is collected, users can leverage the Google Cloud console to:

Logging

gRPC observability provides logs for key RPC events with information to help developers understand the context when these events occur. This contextual information can include which gRPC service/method is being invoked, whether the events happen on the client side or server side, whether it’s sending metadata or payloads, the size of the corresponding data, and even the concrete content of the metadata and/or payloads. These log entries are then presented in Cloud Logging with helpers to filter and even customize the query to search related logs.

\"1

Metrics

gRPC observability provides several metrics: the round trip latency of RPCs, how many RPCs were started and finished during a specific period of time, and even the number of bytes sent/received over the wire. All these metrics can be grouped by a few important parameters, including service/method name and final status. Platform-specific metrics can be included as well, depending on the Google Cloud environment and the gRPC payload actually running. For example, on the Google Kubernetes Engine (GKE) platform, developers can group/filter by namespace, container, and pod information fields to dig into more granular statistical data. With these metrics, Cloud Monitoring enables users to identify problems including:

\"2

Tracing

gRPC observability also allows developers to configure the sampling rate of RPCs. The sampling decision is propagated across the whole system, thus no matter where the RPCs actually happen, developers can always see a complete, end-to-end distributed trace for their processing logic. Sampled RPCs and any further RPCs triggered by them are displayed in Cloud Trace as parent/children spans.

\"3

Getting started

With gRPC observability, telemetry data (logs, metrics, traces) of gRPC workloads can be collected and reported to the Google  Cloud operations suite. It helps developers get a better understanding of their systems and enables them to diagnose problems such as:

To get started with gRPC observability, see our user guide.

", "url": "https://cloud.google.com/blog/products/networking/introducing-grpc-observability-for-microservices/", "title": "Introducing gRPC observability for microservices", "date_modified": "2022-10-21T01:00:00.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/910766/7678f8c4ede60928/", "title": "Identity management for WireGuard networks", "date_modified": "2022-10-19T12:43:56.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/910766/7678f8c4ede60928/", "title": "Identity management for WireGuard", "date_modified": "2022-10-19T12:43:56.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/", "title": "Fine-grained personal access tokens for GitHub", "date_modified": "2022-10-18T15:36:45.000Z" }, { "content_html": "Comments", "url": "https://github.com/e-dant/watcher", "title": "Show HN: Filesystem Watcher", "date_modified": "2022-10-18T13:54:09.000Z" }, { "content_html": "Comments", "url": "https://www.jetstack.io/blog/announcing-paranoia/", "title": "Jetstack Paranoia: A New Open-Source Tool for Container Image Security", "date_modified": "2022-10-18T10:58:03.000Z" }, { "content_html": "Comments", "url": "https://drewdevault.com/2022/10/18/TOTP-is-easy.html", "title": "TOTP for 2FA is incredibly easy to implement. So what's your excuse?", "date_modified": "2022-10-18T10:02:43.000Z" }, { "content_html": "Comments", "url": "https://blog.doyensec.com/2022/10/11/ebpf-bypass-security-monitoring.html", "title": "On Bypassing eBPF Security Monitoring", "date_modified": "2022-10-17T16:00:35.000Z" }, { "content_html": "Comments", "url": "https://notado.substack.com/p/how-flyio-and-tailscale-saved-notado", "title": "How Fly.io and Tailscale Saved Notado", "date_modified": "2022-10-17T15:37:10.000Z" }, { "content_html": "Comments", "url": "https://paulosman.me/2022/10/17/being-on-a-board/", "title": "Being on a Board as an Employee", "date_modified": "2022-10-17T15:28:20.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=33224240", "title": "Ask HN: How to Learn to Sell?", "date_modified": "2022-10-16T15:19:19.000Z" }, { "content_html": "Comments", "url": "https://www.willmcgugan.com/blog/tech/post/ceo-just-wants-to-draw-boxes/", "title": "A possibly new way of drawing boxes in the terminal", "date_modified": "2022-10-15T17:15:01.000Z" }, { "content_html": "Comments", "url": "https://mdwdotla.medium.com/everything-i-wish-i-had-known-about-raising-a-seed-round-a615f8f7740b", "title": "Everything I wish I had known about raising a seed round", "date_modified": "2022-10-11T04:58:22.000Z" }, { "content_html": "Comments", "url": "https://stepci.com/", "title": "Show HN: We built a tool that automatically generates your API Tests", "date_modified": "2022-10-10T13:57:50.000Z" }, { "content_html": "
\"Assembly
\"Assembly

Early on when we learn to program, we get introduced to the concept of recursion. And that it is handy for computing, among other things, sequences defined in terms of recurrences. Such as the famous Fibonnaci numbers - Fn = Fn-1 + Fn-2.

\"Assembly

Later on, perhaps when diving into multithreaded programming, we come to terms with the fact that the stack space for call frames is finite. And that there is an “okay” way and a “cool” way to calculate the Fibonacci numbers using recursion:

// fib_okay.c\n\n#include <stdint.h>\n\nuint64_t fib(uint64_t n)\n{\n        if (n == 0 || n == 1)\n                return 1;\n\n        return fib(n - 1) + fib(n - 2);\n}\n
\n

Listing 1. An okay Fibonacci number generator implementation

\n
// fib_cool.c\n\n#include <stdint.h>\n\nstatic uint64_t fib_tail(uint64_t n, uint64_t a, uint64_t b)\n{\n    if (n == 0)\n        return a;\n    if (n == 1)\n        return b;\n\n    return fib_tail(n - 1, b, a + b);\n}\n\nuint64_t fib(uint64_t n)\n{\n    return fib_tail(n, 1, 1);\n}\n
\n

Listing 2. A better version of the same

\n

If we take a look at the machine code the compiler produces, the “cool” variant translates to a nice and tight sequence of instructions:

⚠ DISCLAIMER: This blog post is assembly-heavy. We will be looking at assembly code for x86-64, arm64 and BPF architectures. If you need an introduction or a refresher, I can recommend “Low-Level Programming” by Igor Zhirkov for x86-64, and “Programming with 64-Bit ARM Assembly Language” by Stephen Smith for arm64. For BPF, see the Linux kernel documentation.

\"Assembly

Listing 3. fib_cool.c compiled for x86-64 and arm64

\n

The “okay” variant, disappointingly, leads to more instructions than a listing can fit. It is a spaghetti of basic blocks.

\"Assembly

But more importantly, it is not free of x86 call instructions.

$ objdump -d fib_okay.o | grep call\n 10c:   e8 00 00 00 00          call   111 <fib+0x111>\n$ objdump -d fib_cool.o | grep call\n$\n
\n

This has an important consequence - as fib recursively calls itself, the stacks keep growing. We can observe it with a bit of help from the debugger.

$ gdb --quiet --batch --command=trace_rsp.gdb --args ./fib_okay 6\nBreakpoint 1 at 0x401188: file fib_okay.c, line 3.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library \"/lib64/libthread_db.so.1\".\nn = 6, %rsp = 0xffffd920\nn = 5, %rsp = 0xffffd900\nn = 4, %rsp = 0xffffd8e0\nn = 3, %rsp = 0xffffd8c0\nn = 2, %rsp = 0xffffd8a0\nn = 1, %rsp = 0xffffd880\nn = 1, %rsp = 0xffffd8c0\nn = 2, %rsp = 0xffffd8e0\nn = 1, %rsp = 0xffffd8c0\nn = 3, %rsp = 0xffffd900\nn = 2, %rsp = 0xffffd8e0\nn = 1, %rsp = 0xffffd8c0\nn = 1, %rsp = 0xffffd900\n13\n[Inferior 1 (process 50904) exited normally]\n$\n
\n

While the “cool” variant makes no use of the stack.

$ gdb --quiet --batch --command=trace_rsp.gdb --args ./fib_cool 6\nBreakpoint 1 at 0x40118a: file fib_cool.c, line 13.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library \"/lib64/libthread_db.so.1\".\nn = 6, %rsp = 0xffffd938\n13\n[Inferior 1 (process 50949) exited normally]\n$\n
\n

Where did the calls go?

\n

The smart compiler turned the last function call in the body into a regular jump. Why was it allowed to do that?

It is the last instruction in the function body we are talking about. The caller stack frame is going to be destroyed right after we return anyway. So why keep it around when we can reuse it for the callee’s stack frame?

This optimization, known as tail call elimination, leaves us with no function calls in the “cool” variant of our fib implementation. There was only one call to eliminate - right at the end.

Once applied, the call becomes a jump (loop). If assembly is not your second language, decompiling the fib_cool.o object file with Ghidra helps see the transformation:

long fib(ulong param_1)\n\n{\n  long lVar1;\n  long lVar2;\n  long lVar3;\n  \n  if (param_1 < 2) {\n    lVar3 = 1;\n  }\n  else {\n    lVar3 = 1;\n    lVar2 = 1;\n    do {\n      lVar1 = lVar3;\n      param_1 = param_1 - 1;\n      lVar3 = lVar2 + lVar1;\n      lVar2 = lVar1;\n    } while (param_1 != 1);\n  }\n  return lVar3;\n}\n
\n

Listing 4. fib_cool.o decompiled by Ghidra

\n

This is very much desired. Not only is the generated machine code much shorter. It is also way faster due to lack of calls, which pop up on the profile for fib_okay.

But I am no performance ninja and this blog post is not about compiler optimizations. So why am I telling you about it?

\"Assembly
Alex Dunkel (Maky), CC BY-SA 3.0, via Wikimedia Commons

Tail calls in BPF

The concept of tail call elimination made its way into the BPF world. Although not in the way you might expect. Yes, the LLVM compiler does get rid of the trailing function calls when building for -target bpf. The transformation happens at the intermediate representation level, so it is backend agnostic. This can save you some BPF-to-BPF function calls, which you can spot by looking for call -N instructions in the BPF assembly.

However, when we talk about tail calls in the BPF context, we usually have something else in mind. And that is a mechanism, built into the BPF JIT compiler, for chaining BPF programs.

We first adopted BPF tail calls when building our XDP-based packet processing pipeline. Thanks to it, we were able to divide the processing logic into several XDP programs. Each responsible for doing one thing.

\"Assembly
Slide from “XDP based DDoS Mitigation” talk by Arthur Fabre

BPF tail calls have served us well since then. But they do have their caveats. Until recently it was impossible to have both BPF tails calls and BPF-to-BPF function calls in the same XDP program on arm64, which is one of the supported architectures for us.

Why? Before we get to that, we have to clarify what a BPF tail call actually does.

A tail call is a tail call is a tail call

BPF exposes the tail call mechanism through the bpf_tail_call helper, which we can invoke from our BPF code. We don’t directly point out which BPF program we would like to call. Instead, we pass it a BPF map (a container) capable of holding references to BPF programs (BPF_MAP_TYPE_PROG_ARRAY), and an index into the map.

long bpf_tail_call(void *ctx, struct bpf_map *prog_array_map, u32 index)\n\n       Description\n              This  special  helper is used to trigger a \"tail call\", or\n              in other words, to jump into  another  eBPF  program.  The\n              same  stack frame is used (but values on stack and in reg‐\n              isters for the caller are not accessible to  the  callee).\n              This  mechanism  allows  for  program chaining, either for\n              raising the maximum number of available eBPF instructions,\n              or  to  execute  given programs in conditional blocks. For\n              security reasons, there is an upper limit to the number of\n              successive tail calls that can be performed.\n
\n

bpf-helpers(7) man page

\n

At first glance, this looks somewhat similar to the execve(2) syscall. It is easy to mistake it for a way to execute a new program from the current program context. To quote the excellent BPF and XDP Reference Guide from the Cilium project documentation:

Tail calls can be seen as a mechanism that allows one BPF program to call another, without returning to the old program. Such a call has minimal overhead as unlike function calls, it is implemented as a long jump, reusing the same stack frame.

But once we add BPF function calls into the mix, it becomes clear that the BPF tail call mechanism is indeed an implementation of tail call elimination, rather than a way to replace one program with another:

Tail calls, before the actual jump to the target program, will unwind only its current stack frame. As we can see in the example above, if a tail call occurs from within the sub-function, the function’s (func1) stack frame will be present on the stack when a program execution is at func2. Once the final function (func3) function terminates, all the previous stack frames will be unwinded and control will get back to the caller of BPF program caller.

Alas, one with sometimes slightly surprising semantics. Consider the code like below, where a BPF function calls the bpf_tail_call() helper:

struct {\n    __uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n    __uint(max_entries, 1);\n    __uint(key_size, sizeof(__u32));\n    __uint(value_size, sizeof(__u32));\n} bar SEC(\".maps\");\n\nSEC(\"tc\")\nint serve_drink(struct __sk_buff *skb __unused)\n{\n    return 0xcafe;\n}\n\nstatic __noinline\nint bring_order(struct __sk_buff *skb)\n{\n    bpf_tail_call(skb, &bar, 0);\n    return 0xf00d;\n}\n\nSEC(\"tc\")\nint server1(struct __sk_buff *skb)\n{\n    return bring_order(skb);    \n}\n\nSEC(\"tc\")\nint server2(struct __sk_buff *skb)\n{\n    __attribute__((musttail)) return bring_order(skb);  \n}\n
\n

We have two seemingly not so different BPF programs - server1() and server2(). They both call the same BPF function bring_order(). The function tail calls into the serve_drink() program, if the bar[0] map entry points to it (let’s assume that).

Do both server1 and server2 return the same value? Turns out that - no, they don’t. We get a hex 🍔 from server1, and a ☕ from server2. How so?

First thing to notice is that a BPF tail call unwinds just the current function stack frame. Code past the bpf_tail_call() invocation in the function body never executes, providing the tail call is successful (the map entry was set, and the tail call limit has not been reached).

When the tail call finishes, control returns to the caller of the function which made the tail call. Applying this to our example, the control flow is serverX() --> bring_order() --> bpf_tail_call() --> serve_drink() -return-> serverX() for both programs.

The second thing to keep in mind is that the compiler does not know that the bpf_tail_call() helper changes the control flow. Hence, the unsuspecting compiler optimizes the code as if the execution would continue past the BPF tail call.

\"Assembly
The call graph for server1() and server2() is the same, but the return value differs due to build time optimizations.

In our case, the compiler thinks it is okay to propagate the constant which bring_order() returns to server1(). Possibly catching us by surprise, if we didn’t check the generated BPF assembly.

We can prevent it by forcing the compiler to make a tail call to bring_order(). This way we ensure that whatever bring_order() returns will be used as the server2() program result.

🛈 General rule - for least surprising results, use musttail attribute when calling a function that contain a BPF tail call.

How does the bpf_tail_call() work underneath then? And why the BPF verifier wouldn’t let us mix the function calls with tail calls on arm64? Time to dig deeper.

\"Assembly
Public Domain image

BPF tail call on x86-64

What does a bpf_tail_call() helper call translate to after BPF JIT for x86-64 has compiled it? How does the implementation guarantee that we don’t end up in a tail call loop forever?

To find out we will need to piece together a few things.

First, there is the BPF JIT compiler source code, which lives in arch/x86/net/bpf_jit_comp.c. Its code is annotated with helpful comments. We will focus our attention on the following call chain within the JIT:

do_jit() 🔗
   emit_prologue() 🔗
   push_callee_regs() 🔗
   for (i = 1; i <= insn_cnt; i++, insn++) {
     switch (insn->code) {
     case BPF_JMP | BPF_CALL:
       /* emit function call */ 🔗
     case BPF_JMP | BPF_TAIL_CALL:
       emit_bpf_tail_call_direct() 🔗
     case BPF_JMP | BPF_EXIT:
       /* emit epilogue */ 🔗
     }
  }

It is sometimes hard to visualize the generated instruction stream just from reading the compiler code. Hence, we will also want to inspect the input - BPF instructions - and the output - x86-64 instructions - of the JIT compiler.

To inspect BPF and x86-64 instructions of a loaded BPF program, we can use bpftool prog dump. However, first we must populate the BPF map used as the tail call jump table. Otherwise, we might not be able to see the tail call jump!

This is due to optimizations that use instruction patching when the index into the program array is known at load time.

# bpftool prog loadall ./tail_call_ex1.o /sys/fs/bpf pinmaps /sys/fs/bpf\n# bpftool map update pinned /sys/fs/bpf/jmp_table key 0 0 0 0 value pinned /sys/fs/bpf/target_prog\n# bpftool prog dump xlated pinned /sys/fs/bpf/entry_prog\nint entry_prog(struct __sk_buff * skb):\n; bpf_tail_call(skb, &jmp_table, 0);\n   0: (18) r2 = map[id:24]\n   2: (b7) r3 = 0\n   3: (85) call bpf_tail_call#12\n; return 0xf00d;\n   4: (b7) r0 = 61453\n   5: (95) exit\n# bpftool prog dump jited pinned /sys/fs/bpf/entry_prog\nint entry_prog(struct __sk_buff * skb):\nbpf_prog_4f697d723aa87765_entry_prog:\n; bpf_tail_call(skb, &jmp_table, 0);\n   0:   nopl   0x0(%rax,%rax,1)\n   5:   xor    %eax,%eax\n   7:   push   %rbp\n   8:   mov    %rsp,%rbp\n   b:   push   %rax\n   c:   movabs $0xffff888102764800,%rsi\n  16:   xor    %edx,%edx\n  18:   mov    -0x4(%rbp),%eax\n  1e:   cmp    $0x21,%eax\n  21:   jae    0x0000000000000037\n  23:   add    $0x1,%eax\n  26:   mov    %eax,-0x4(%rbp)\n  2c:   nopl   0x0(%rax,%rax,1)\n  31:   pop    %rax\n  32:   jmp    0xffffffffffffffe3   // bug? 🤔\n; return 0xf00d;\n  37:   mov    $0xf00d,%eax\n  3c:   leave\n  3d:   ret\n
\n

There is a caveat. The target addresses for tail call jumps in bpftool prog dump jited output will not make any sense. To discover the real jump targets, we have to peek into the kernel memory. That can be done with gdb after we find the address of our JIT’ed BPF programs in /proc/kallsyms:

# tail -2 /proc/kallsyms\nffffffffa0000720 t bpf_prog_f85b2547b00cbbe9_target_prog        [bpf]\nffffffffa0000748 t bpf_prog_4f697d723aa87765_entry_prog [bpf]\n# gdb -q -c /proc/kcore -ex 'x/18i 0xffffffffa0000748' -ex 'quit'\n[New process 1]\nCore was generated by `earlyprintk=serial,ttyS0,115200 console=ttyS0 psmouse.proto=exps \"virtme_stty_c'.\n#0  0x0000000000000000 in ?? ()\n   0xffffffffa0000748:  nopl   0x0(%rax,%rax,1)\n   0xffffffffa000074d:  xor    %eax,%eax\n   0xffffffffa000074f:  push   %rbp\n   0xffffffffa0000750:  mov    %rsp,%rbp\n   0xffffffffa0000753:  push   %rax\n   0xffffffffa0000754:  movabs $0xffff888102764800,%rsi\n   0xffffffffa000075e:  xor    %edx,%edx\n   0xffffffffa0000760:  mov    -0x4(%rbp),%eax\n   0xffffffffa0000766:  cmp    $0x21,%eax\n   0xffffffffa0000769:  jae    0xffffffffa000077f\n   0xffffffffa000076b:  add    $0x1,%eax\n   0xffffffffa000076e:  mov    %eax,-0x4(%rbp)\n   0xffffffffa0000774:  nopl   0x0(%rax,%rax,1)\n   0xffffffffa0000779:  pop    %rax\n   0xffffffffa000077a:  jmp    0xffffffffa000072b\n   0xffffffffa000077f:  mov    $0xf00d,%eax\n   0xffffffffa0000784:  leave\n   0xffffffffa0000785:  ret\n# gdb -q -c /proc/kcore -ex 'x/7i 0xffffffffa0000720' -ex 'quit'\n[New process 1]\nCore was generated by `earlyprintk=serial,ttyS0,115200 console=ttyS0 psmouse.proto=exps \"virtme_stty_c'.\n#0  0x0000000000000000 in ?? ()\n   0xffffffffa0000720:  nopl   0x0(%rax,%rax,1)\n   0xffffffffa0000725:  xchg   %ax,%ax\n   0xffffffffa0000727:  push   %rbp\n   0xffffffffa0000728:  mov    %rsp,%rbp\n   0xffffffffa000072b:  mov    $0xcafe,%eax\n   0xffffffffa0000730:  leave\n   0xffffffffa0000731:  ret\n#\n
\n

Lastly, it will be handy to have a cheat sheet of mapping between BPF registers (r0, r1, …) to hardware registers (rax, rdi, …) that the JIT compiler uses.

\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
BPFx86-64
r0rax
r1rdi
r2rsi
r3rdx
r4rcx
r5r8
r6rbx
r7r13
r8r14
r9r15
r10rbp
internalr9-r12

Now we are prepared to work out what happens when we use a BPF tail call.

\"Assembly

In essence, bpf_tail_call() emits a jump into another function, reusing the current stack frame. It is just like a regular optimized tail call, but with a twist.

Because of the BPF security guarantees - execution terminates, no stack overflows - there is a limit on the number of tail calls we can have (MAX_TAIL_CALL_CNT = 33).

Counting the tail calls across BPF programs is not something we can do at load-time. The jump table (BPF program array) contents can change after the program has been verified. Our only option is to keep track of tail calls at run-time. That is why the JIT’ed code for the bpf_tail_call() helper checks and updates the tail_call_cnt counter.

The updated count is then passed from one BPF program to another, and from one BPF function to another, as we will see, through the rax register (r0 in BPF).

Luckily for us, the x86-64 calling convention dictates that the rax register does not partake in passing function arguments, but rather holds the function return value. The JIT can repurpose it to pass an additional - hidden - argument.

The function body is, however, free to make use of the r0/rax register in any way it pleases. This explains why we want to save the tail_call_cnt passed via rax onto stack right after we jump to another program. bpf_tail_call() can later load the value from a known location on the stack.

This way, the code emitted for each bpf_tail_call() invocation, and the BPF function prologue work in tandem, keeping track of tail call count across BPF program boundaries.

But what if our BPF program is split up into several BPF functions, each with its own stack frame? What if these functions perform BPF tail calls? How is the tail call count tracked then?

Mixing BPF function calls with BPF tail calls

BPF has its own terminology when it comes to functions and calling them, which is influenced by the internal implementation. Function calls are referred to as BPF to BPF calls. Also, the main/entry function in your BPF code is called “the program”, while all other functions are known as “subprograms”.

Each call to subprogram allocates a stack frame for local state, which persists until the function returns. Naturally, BPF subprogram calls can be nested creating a call chain. Just like nested function calls in user-space.

BPF subprograms are also allowed to make BPF tail calls. This, effectively, is a mechanism for extending the call chain to another BPF program and its subprograms.

If we cannot track how long the call chain can be, and how much stack space each function uses, we put ourselves at risk of overflowing the stack. We cannot let this happen, so BPF enforces limitations on when and how many BPF tail calls can be done:

static int check_max_stack_depth(struct bpf_verifier_env *env)\n{\n        …\n        /* protect against potential stack overflow that might happen when\n         * bpf2bpf calls get combined with tailcalls. Limit the caller's stack\n         * depth for such case down to 256 so that the worst case scenario\n         * would result in 8k stack size (32 which is tailcall limit * 256 =\n         * 8k).\n         *\n         * To get the idea what might happen, see an example:\n         * func1 -> sub rsp, 128\n         *  subfunc1 -> sub rsp, 256\n         *  tailcall1 -> add rsp, 256\n         *   func2 -> sub rsp, 192 (total stack size = 128 + 192 = 320)\n         *   subfunc2 -> sub rsp, 64\n         *   subfunc22 -> sub rsp, 128\n         *   tailcall2 -> add rsp, 128\n         *    func3 -> sub rsp, 32 (total stack size 128 + 192 + 64 + 32 = 416)\n         *\n         * tailcall will unwind the current stack frame but it will not get rid\n         * of caller's stack as shown on the example above.\n         */\n        if (idx && subprog[idx].has_tail_call && depth >= 256) {\n                verbose(env,\n                        \"tail_calls are not allowed when call stack of previous frames is %d bytes. Too large\\n\",\n                        depth);\n                return -EACCES;\n        }\n        …\n}\n
\n

While the stack depth can be calculated by the BPF verifier at load-time, we still need to keep count of tail call jumps at run-time. Even when subprograms are involved.

This means that we have to pass the tail call count from one BPF subprogram to another, just like we did when making a BPF tail call, so we yet again turn to value passing through the rax register.

\"Assembly
Control flow in a BPF program with a function call followed by a tail call.

🛈 To keep things simple, BPF code in our examples does not allocate anything on stack. I encourage you to check how the JIT’ed code changes when you add some local variables. Just make sure the compiler does not optimize them out.

To make it work, we need to:

① load the tail call count saved on stack into rax before call’ing the subprogram,
② adjust the subprogram prologue, so that it does not reset the rax like the main program does,
③ save the passed tail call count on subprogram’s stack for the bpf_tail_call() helper to consume it.

A bpf_tail_call() within our suprogram will then:

④ load the tail call count from stack,
⑤ unwind the BPF stack, but keep the current subprogram’s stack frame in tact, and
⑥ jump to the target BPF program.

Now we have seen how all the pieces of the puzzle fit together to make BPF tail work on x86-64 safely. The only open question is does it work the same way on other platforms like arm64? Time to shift gears and dive into a completely different BPF JIT implementation.

\"Assembly
Based on an image by Wutthichai Charoenburi, CC BY 2.0

Tail calls on arm64

If you try loading a BPF program that uses both BPF function calls (aka BPF to BPF calls) and BPF tail calls on an arm64 machine running the latest 5.15 LTS kernel, or even the latest 5.19 stable kernel, the BPF verifier will kindly ask you to reconsider your choice:

# uname -rm\n5.19.12 aarch64\n# bpftool prog loadall tail_call_ex2.o /sys/fs/bpf\nlibbpf: prog 'entry_prog': BPF program load failed: Invalid argument\nlibbpf: prog 'entry_prog': -- BEGIN PROG LOAD LOG --\n0: R1=ctx(off=0,imm=0) R10=fp0\n; __attribute__((musttail)) return sub_func(skb);\n0: (85) call pc+1\ncaller:\n R10=fp0\ncallee:\n frame1: R1=ctx(off=0,imm=0) R10=fp0\n; bpf_tail_call(skb, &jmp_table, 0);\n2: (18) r2 = 0xffffff80c38c7200       ; frame1: R2_w=map_ptr(off=0,ks=4,vs=4,imm=0)\n4: (b7) r3 = 0                        ; frame1: R3_w=P0\n5: (85) call bpf_tail_call#12\ntail_calls are not allowed in non-JITed programs with bpf-to-bpf calls\nprocessed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n-- END PROG LOAD LOG --\n…\n#\n
\n

That is a pity! We have been looking forward to reaping the benefits of code sharing with BPF to BPF calls in our lengthy machine generated BPF programs. So we asked - how hard could it be to make it work?

After all, BPF JIT for arm64 already can handle BPF tail calls and BPF to BPF calls, when used in isolation.

It is “just” a matter of understanding the existing JIT implementation, which lives in arch/arm64/net/bpf_jit_comp.c, and identifying the missing pieces.

To understand how BPF JIT for arm64 works, we will use the same method as before - look at its code together with sample input (BPF instructions) and output (arm64 instructions).

We don’t have to read the whole source code. It is enough to zero in on a few particular code paths:

bpf_int_jit_compile() 🔗
   build_prologue() 🔗
   build_body() 🔗
     for (i = 0; i < prog->len; i++) {
        build_insn() 🔗
          switch (code) {
          case BPF_JMP | BPF_CALL:
            /* emit function call */ 🔗
          case BPF_JMP | BPF_TAIL_CALL:
            emit_bpf_tail_call() 🔗
          }
     }
   build_epilogue() 🔗

One thing that the arm64 architecture, and RISC architectures in general, are known for is that it has a plethora of general purpose registers (x0-x30). This is a good thing. We have more registers to allocate to JIT internal state, like the tail call count. A cheat sheet of what roles the hardware registers play in the BPF JIT will be helpful:

\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n
BPFarm64
r0x7
r1x0
r2x1
r3x2
r4x3
r5x4
r6x19
r7x20
r8x21
r9x22
r10x25
internalx9-x12, x26 (tail_call_cnt), x27

Now let’s try to understand the state of things by looking at the JIT’s input and output for two particular scenarios: (1) a BPF tail call, and (2) a BPF to BPF call.

It is hard to read assembly code selectively. We will have to go through all instructions one by one, and understand what each one is doing.

⚠ Brace yourself. Time to decipher a bit of ARM64 assembly. If this will be your first time reading ARM64 assembly, you might want to at least skim through this Guide to ARM64 / AArch64 Assembly on Linux before diving in.

Scenario #1: A single BPF tail call - tail_call_ex1.bpf.c

Input: BPF assembly (bpftool prog dump xlated)

   0: (18) r2 = map[id:4]           // jmp_table map\n   2: (b7) r3 = 0\n   3: (85) call bpf_tail_call#12\n   4: (b7) r0 = 61453               // 0xf00d\n   5: (95) exit\n
\n

Output: ARM64 assembly (bpftool prog dump jited)

 0:   paciasp                            // Sign LR (ROP protection) ①\n 4:   stp     x29, x30, [sp, #-16]!      // Save FP and LR registers ②\n 8:   mov     x29, sp                    // Set up Frame Pointer\n c:   stp     x19, x20, [sp, #-16]!      // Save callee-saved registers ③\n10:   stp     x21, x22, [sp, #-16]!      // ⋮ \n14:   stp     x25, x26, [sp, #-16]!      // ⋮ \n18:   stp     x27, x28, [sp, #-16]!      // ⋮ \n1c:   mov     x25, sp                    // Set up BPF stack base register (r10)\n20:   mov     x26, #0x0                  // Initialize tail_call_cnt ④\n24:   sub     x27, x25, #0x0             // Calculate FP bottom ⑤\n28:   sub     sp, sp, #0x200             // Set up BPF program stack ⑥\n2c:   mov     x1, #0xffffff80ffffffff    // r2 = map[id:4] ⑦\n30:   movk    x1, #0xc38c, lsl #16       // ⋮ \n34:   movk    x1, #0x7200                // ⋮\n38:   mov     x2, #0x0                   // r3 = 0\n3c:   mov     w10, #0x24                 // = offsetof(struct bpf_array, map.max_entries) ⑧\n40:   ldr     w10, [x1, x10]             // Load array->map.max_entries\n44:   add     w2, w2, #0x0               // = index (0)\n48:   cmp     w2, w10                    // if (index >= array->map.max_entries)\n4c:   b.cs    0x0000000000000088         //     goto out;\n50:   mov     w10, #0x21                 // = MAX_TAIL_CALL_CNT (33)\n54:   cmp     x26, x10                   // if (tail_call_cnt >= MAX_TAIL_CALL_CNT)\n58:   b.cs    0x0000000000000088         //     goto out;\n5c:   add     x26, x26, #0x1             // tail_call_cnt++;\n60:   mov     w10, #0x110                // = offsetof(struct bpf_array, ptrs)\n64:   add     x10, x1, x10               // = &array->ptrs\n68:   lsl     x11, x2, #3                // = index * sizeof(array->ptrs[0])\n6c:   ldr     x11, [x10, x11]            // prog = array->ptrs[index];\n70:   cbz     x11, 0x0000000000000088    // if (prog == NULL) goto out;\n74:   mov     w10, #0x30                 // = offsetof(struct bpf_prog, bpf_func)\n78:   ldr     x10, [x11, x10]            // Load prog->bpf_func\n7c:   add     x10, x10, #0x24            // += PROLOGUE_OFFSET * AARCH64_INSN_SIZE (4)\n80:   add     sp, sp, #0x200             // Unwind BPF stack\n84:   br      x10                        // goto *(prog->bpf_func + prologue_offset)\n88:   mov     x7, #0xf00d                // r0 = 0xf00d\n8c:   add     sp, sp, #0x200             // Unwind BPF stack ⑨\n90:   ldp     x27, x28, [sp], #16        // Restore used callee-saved registers\n94:   ldp     x25, x26, [sp], #16        // ⋮\n98:   ldp     x21, x22, [sp], #16        // ⋮\n9c:   ldp     x19, x20, [sp], #16        // ⋮\na0:   ldp     x29, x30, [sp], #16        // ⋮\na4:   add     x0, x7, #0x0               // Set return value\na8:   autiasp                            // Authenticate LR\nac:   ret                                // Return to caller\n
\n

① BPF program prologue starts with Pointer Authentication Code (PAC), which protects against Return Oriented Programming attacks. PAC instructions are emitted by JIT only if CONFIG_ARM64_PTR_AUTH_KERNEL is enabled.

② Arm 64 Architecture Procedure Call Standard mandates that the Frame Pointer (register X29) and the Link Register (register X30), aka the return address, of the caller should be recorded onto the stack.

③ Registers X19 to X28, and X29 (FP) plus X30 (LR), are callee saved. ARM64 BPF JIT does not use registers X23 and X24 currently, so they are not saved.

④ We track the tail call depth in X26. No need to save it onto stack since we use a register dedicated just for this purpose.

⑤ FP bottom is an optimization that allows store/loads to BPF stack with a single instruction and an immediate offset value.

⑥ Reserve space for the BPF program stack. The stack layout is now as shown in a diagram in build_prologue() source code.

⑦ The BPF function body starts here.

⑧ bpf_tail_call() instructions start here.

⑨ The epilogue starts here.

Whew! That was a handful 😅.

Notice that the BPF tail call implementation on arm64 is not as optimized as on x86-64. There is no code patching to make direct jumps when the target program index is known at the JIT-compilation time. Instead, the target address is always loaded from the BPF program array.

Ready for the second scenario? I promise it will be shorter. Function prologue and epilogue instructions will look familiar, so we are going to keep annotations down to a minimum.

Scenario #2: A BPF to BPF call - sub_call_ex1.bpf.c

Input: BPF assembly (bpftool prog dump xlated)

int entry_prog(struct __sk_buff * skb):\n   0: (85) call pc+1#bpf_prog_a84919ecd878b8f3_sub_func\n   1: (95) exit\nint sub_func(struct __sk_buff * skb):\n   2: (b7) r0 = 61453                   // 0xf00d\n   3: (95) exit\n
\n

Output: ARM64 assembly

int entry_prog(struct __sk_buff * skb):\nbpf_prog_163e74e7188910f2_entry_prog:\n   0:   paciasp                                 // Begin prologue\n   4:   stp     x29, x30, [sp, #-16]!           // ⋮\n   8:   mov     x29, sp                         // ⋮\n   c:   stp     x19, x20, [sp, #-16]!           // ⋮\n  10:   stp     x21, x22, [sp, #-16]!           // ⋮\n  14:   stp     x25, x26, [sp, #-16]!           // ⋮\n  18:   stp     x27, x28, [sp, #-16]!           // ⋮\n  1c:   mov     x25, sp                         // ⋮\n  20:   mov     x26, #0x0                       // ⋮\n  24:   sub     x27, x25, #0x0                  // ⋮\n  28:   sub     sp, sp, #0x0                    // End prologue\n  2c:   mov     x10, #0xffffffffffff5420        // Build sub_func()+0x0 address\n  30:   movk    x10, #0x8ff, lsl #16            // ⋮\n  34:   movk    x10, #0xffc0, lsl #32           // ⋮\n  38:   blr     x10 ------------------.         // Call sub_func()+0x0 \n  3c:   add     x7, x0, #0x0 <----------.       // r0 = sub_func()\n  40:   mov     sp, sp                | |       // Begin epilogue\n  44:   ldp     x27, x28, [sp], #16   | |       // ⋮\n  48:   ldp     x25, x26, [sp], #16   | |       // ⋮\n  4c:   ldp     x21, x22, [sp], #16   | |       // ⋮\n  50:   ldp     x19, x20, [sp], #16   | |       // ⋮\n  54:   ldp     x29, x30, [sp], #16   | |       // ⋮\n  58:   add     x0, x7, #0x0          | |       // ⋮\n  5c:   autiasp                       | |       // ⋮\n  60:   ret                           | |       // End epilogue\n                                      | |\nint sub_func(struct __sk_buff * skb): | |\nbpf_prog_a84919ecd878b8f3_sub_func:   | |\n   0:   paciasp <---------------------' |       // Begin prologue\n   4:   stp     x29, x30, [sp, #-16]!   |       // ⋮\n   8:   mov     x29, sp                 |       // ⋮\n   c:   stp     x19, x20, [sp, #-16]!   |       // ⋮\n  10:   stp     x21, x22, [sp, #-16]!   |       // ⋮\n  14:   stp     x25, x26, [sp, #-16]!   |       // ⋮\n  18:   stp     x27, x28, [sp, #-16]!   |       // ⋮\n  1c:   mov     x25, sp                 |       // ⋮\n  20:   mov     x26, #0x0               |       // ⋮\n  24:   sub     x27, x25, #0x0          |       // ⋮\n  28:   sub     sp, sp, #0x0            |       // End prologue\n  2c:   mov     x7, #0xf00d             |       // r0 = 0xf00d\n  30:   mov     sp, sp                  |       // Begin epilogue\n  34:   ldp     x27, x28, [sp], #16     |       // ⋮\n  38:   ldp     x25, x26, [sp], #16     |       // ⋮\n  3c:   ldp     x21, x22, [sp], #16     |       // ⋮\n  40:   ldp     x19, x20, [sp], #16     |       // ⋮\n  44:   ldp     x29, x30, [sp], #16     |       // ⋮\n  48:   add     x0, x7, #0x0            |       // ⋮\n  4c:   autiasp                         |       // ⋮\n  50:   ret ----------------------------'       // End epilogue\n
\n

We have now seen what a BPF tail call and a BPF function/subprogram call compiles down to. Can you already spot what would go wrong if mixing the two was allowed?

That’s right! Every time we enter a BPF subprogram, we reset the X26 register, which holds the tail call count, to zero (mov x26, #0x0). This is bad. It would let users create program chains longer than the MAX_TAIL_CALL_CNT limit.

How about we just skip this step when emitting the prologue for BPF subprograms?

@@ -246,6 +246,7 @@ static bool is_lsi_offset(int offset, int scale)\n static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)\n {\n        const struct bpf_prog *prog = ctx->prog;\n+       const bool is_main_prog = prog->aux->func_idx == 0;\n        const u8 r6 = bpf2a64[BPF_REG_6];\n        const u8 r7 = bpf2a64[BPF_REG_7];\n        const u8 r8 = bpf2a64[BPF_REG_8];\n@@ -299,7 +300,7 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)\n        /* Set up BPF prog stack base register */\n        emit(A64_MOV(1, fp, A64_SP), ctx);\n\n-       if (!ebpf_from_cbpf) {\n+       if (!ebpf_from_cbpf && is_main_prog) {\n                /* Initialize tail_call_cnt */\n                emit(A64_MOVZ(1, tcc, 0, 0), ctx);\n
\n

Believe it or not. This is everything that was missing to get BPF tail calls working with function calls on arm64. The feature will be enabled in the upcoming Linux 6.0 release.

Outro

From recursion to tweaking the BPF JIT. How did we get here? Not important. It’s all about the journey.

Along the way we have unveiled a few secrets behind BPF tails calls, and hopefully quenched your thirst for low-level programming. At least for today.

All that is left is to sit back and watch the fruits of our work. With GDB hooked up to a VM, we can observe how a BPF program calls into a BPF function, and from there tail calls to another BPF program:

demo-gdb-step-thru-bpf.pages.dev/

Until next time 🖖.

", "url": "https://blog.cloudflare.com/assembly-within-bpf-tail-calls-on-x86-and-arm/", "title": "Assembly within! BPF tail calls on x86 and ARM", "date_modified": "2022-10-10T13:00:00.000Z" }, { "content_html": "Comments", "url": "https://reflame.app/?source=show-hn-launch", "title": "Show HN: Reflame – Deploy your React web apps in milliseconds", "date_modified": "2022-10-08T17:10:57.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/the-future-of-web-is-on-the-edge", "title": "The Future of the Web Is on the Edge", "date_modified": "2022-10-06T17:08:50.000Z" }, { "content_html": "Comments", "url": "https://blog.meadsteve.dev/docker/2022/09/29/docker-build-caching/", "title": "Using a Docker registry as a distributed layer cache for CI", "date_modified": "2022-09-29T11:49:44.000Z" }, { "content_html": "Comments", "url": "https://depot.dev", "title": "Show HN: Depot – fast, remote Docker container builds", "date_modified": "2022-09-28T18:01:01.000Z" }, { "content_html": "Comments", "url": "https://www.inngest.com/blog/modern-serverless-job-scheduler", "title": "Modern Serverless Job Schedulers", "date_modified": "2022-09-28T17:53:24.000Z" }, { "content_html": "\"Leading

This post is also available in 简体中文, 日本語 and Deutsch.

\n
\"Leading

From our earliest days, Cloudflare has stood for helping build a better Internet that’s accessible to all. It’s core to our mission that anyone who wants to start building on the Internet should be able to do so easily, and without the barriers of prohibitively expensive or difficult to use infrastructure.

Nowhere is this philosophy more important – and more impactful to the Internet – than with our developer platform, Cloudflare Workers. Workers is, quite simply, where developers and entrepreneurs start on Day 1. It’s a full developer platform that includes cloud storage; website hosting; SQL databases; and of course, the industry’s leading serverless product. The platform’s ease-of-use and accessible pricing (all the way down to free) are critical in advancing our mission. For startups, this translates into fast, easy deployment and iteration, that scales seamlessly with predictable, transparent and cost-effective pricing. Building a great business from scratch is hard enough – we ought to know! – and so we’re aiming to take all the complexity out of your application infrastructure.

Announcing the Workers Launchpad funding program

Today, we’re taking things a step further and making it easier for startups to build the business of their dreams. We’re announcing a $1.25 billion Workers Launchpad funding program in partnership with some of the world’s leading venture capital firms. Any startup built on Workers can apply. As is the case with the Workers Platform itself, we’ve tried to make applying dead simple: it should take you less than five minutes to submit your application through the Workers Launchpad portal.

How does it work? The only requirement for being eligible for the funding program is that you’ve built your core infrastructure on Workers. If you’re new to Cloudflare and Cloudflare Workers, check out our Startup Plan to get started. We hope these resources will be helpful to all startups and help level the playing field, no matter where in the world you might be.

Once you submit your application, it will be reviewed by our Launchpad team, several of whom are former entrepreneurs and venture capital folks themselves. They’ll match promising applicants with our VC partners who have the most expertise in your space (more on them below). Every quarter, we’ll announce the winners of our Launchpad program. Winners, our “Workers Founders”, will be guaranteed the opportunity to pitch the VC partner(s) that we’ve determined would be a good match for your business. It’s a win-win all around. VCs get the opportunity to invest in businesses they know are being built on a forward-looking, world-class, development platform. Entrepreneurs get connected to world-class VCs. And for the first class of winners, we’ll have a few added perks that we describe in more detail below.

Who are the VCs that you might get a chance to pitch to?

When we approached our friends in the venture community with our vision for the Workers Launchpad, we received incredibly positive feedback and excitement. Many have seen firsthand the competitive advantages of building on Workers through their own portfolio companies. Moreover, Cloudflare is home to one of the largest developer communities on Earth with approximately 20% of the world’s websites on our network. As such, we can play a unique role in matching great entrepreneurs with great VCs to further not only the Workers platform, but also the Internet ecosystem, for everyone.

We’re honored to announce a world-class group of VC Launch Partners supporting this program and the ecosystem of Workers-based startups:

\"Leading

More on why we’re doing this

So why are we doing this? The simple answer is we’re proud of our Workers Developer Platform and think that everyone should be using it. Entrepreneurs who develop on Workers can ship faster; more easily and cost-effectively; and in a way that future proofs your infrastructure:

Speed. Development velocity isn’t just a convenience for an entrepreneur. It’s a massive competitive advantage. In fact, development velocity is one of Cloudflare’s competitive advantages – we’re able to develop quickly because we build on Workers. When you develop on Workers, you don’t need to spend time configuring DNS records, maintaining certificates, scaling up clusters, or building complex deployment pipelines. Focus on developing your application, and Cloudflare will handle the rest.

Ease of use. Startup teams and founders are some of the busiest people on earth. You shouldn’t have to think about – or make complicated decisions about – IT infrastructure. Questions like: “Which availability zone should I choose?”, or “Will I be able to scale up my infrastructure in time for our next viral marketing campaign?” shouldn’t have to cross your mind! And on the Workers Platform, they don’t. The code you and your team writes automatically deploys quickly and consistently across Cloudflare’s global network in 275+ cities in over 100 countries. Cloudflare securely and scalably connects your users to your applications, regardless of where those applications are hosted or how many users suddenly sign up for your product. Developers can easily manage globally distributed applications with a programmable network that easily connects to whatever services they need to talk to.

Future-proofing your infrastructure and your wallet. Cloudflare’s massive global network – that’s distributed across 275+ cities in over 100 countries – is able to scale with your business, no matter how large it grows to become. We also help you remain compliant with local laws and regulations as you expand around the world, with capabilities like Workers’ Jurisdictional Restrictions for Durable Objects. You can sleep soundly at night instead of worrying about how to level up your infrastructure in the midst of shifting regulations, and equally importantly, knowing that you will not wake up to any surprise bills. Many of us have had the experience of being charged unexpected and / or exorbitant fees from our cloud providers. For example, providers will often make it easy and free to onboard your application or data, but charge exorbitant rates when you want to move them out (i.e. egress fees). Cloudflare will never charge for egress. Our pricing is simple, and we constantly aim to be the low-cost provider, no matter how large your business grows to be.

\"Leading

Dogfooding our own product

We’re excited about Workers not only because we’ve built our own infrastructure on it, but also because we’re seeing the incredible things others have built on it. In fact, we acquired a company built entirely on Workers at the end of last year, an Israeli start-up named Zaraz, which secures and accelerates third party web tools. Workers allowed Zaraz to replace the multiple network requests of each tool running on a website with one single request, effectively streamlining a messy web of extensions into a single lightweight application. This acquisition opened our eyes to the power of the global community that’s built on our platform, and left us motivated to help startups built on Workers find the funding, mentorship, and support needed to grow.

But wait, there’s more!

To make it even easier for startups to take advantage of all the benefits that Workers has to offer, applicants to the Workers Launchpad program who have raised less than $3 million in total external funding will automatically have the option to receive Cloudflare’s Startup Plan. This plan includes all the elements of Cloudflare’s Pro and Business Plans ($2,400 annual value) plus higher tiers of our Stream video product, our Teams Zero Trust security suite and the Workers platform. To make sure the full range of our developer platform is accessible to startups, we recently more than tripled the number of products available in this plan, which now includes email security, R2, Pages, KV, and many others.

Furthermore, all startups that apply by October 31, 2022, will be eligible to be selected for the Winter 2022 class of Workers Founders, which will unlock additional support, mentorship, and marketing opportunities. Being selected as a Workers Founder will get you a chance to practice your pitch with investors, engage with leaders from Cloudflare, and get advice on how to build a successful business from topics like recruiting to marketing, sales, and beyond during a virtual Workers Founders Bootcamp Week. The program will culminate in a virtual Demo Day, so you can show the world what you’ve been building. We’re leaning in to help promising entrepreneurs join us in our mission to help build a better Internet.

Helping make the Internet better for all

Accessibility and ease of use are core to everything we do at Cloudflare. We will always make our products and platforms so easy to use that even the smallest business or hobbyist can easily use them. We hope the Workers Launchpad funding program encourages entrepreneurs from all around the world, and from all backgrounds, to start building on Workers, and makes it easier for you to find the funding you need to build the business of your dreams.

Head to the Workers Launchpad page to apply and join the Cloudflare Developer Discord to engage with the Workers community. If you’re a VC that is interested in supporting the program, reach out to Workers-Launchpad@cloudflare.com.

Cloudflare is not providing any funding or making any funding decisions, and there is no guarantee that any particular company will receive funding through the program. All funding decisions will be made by the venture capital firms that participate in the program. Cloudflare is not a registered broker-dealer, investment adviser, or other similar intermediary.

", "url": "https://blog.cloudflare.com/workers-launchpad/", "title": "Leading venture capital firms to provide up to $1.25 BILLION to back startups built on Cloudflare Workers", "date_modified": "2022-09-27T13:00:00.000Z" }, { "content_html": "Comments", "url": "https://a16z.com/2014/05/30/selling-saas-products-dont-sell-themselves/", "title": "If SaaS Products Sell Themselves, Why Do We Need Sales?", "date_modified": "2022-09-27T06:36:38.000Z" }, { "content_html": "Comments", "url": "https://github.com/weaveworks/ignite", "title": "Ignite – Use Firecracker VMs with Docker images", "date_modified": "2022-09-26T23:50:13.000Z" }, { "content_html": "Comments", "url": "https://notes.crmarsh.com/isolates-microvms-and-webassembly", "title": "Isolates, microVMs, and WebAssembly", "date_modified": "2022-09-26T20:20:15.000Z" }, { "content_html": "Comments", "url": "https://nanovms.com/dev/tutorials/cross-region-unikernels-edge", "title": "Cross Region Unikernels at the Edge", "date_modified": "2022-09-26T18:46:58.000Z" }, { "content_html": "Comments", "url": "https://www.bytebase.com/blog/30saas-services-behind-startup", "title": "Bytebase: 20-Person Startup, 30 SaaS Services, and $1,183 Monthly Bill", "date_modified": "2022-09-26T12:47:00.000Z" }, { "content_html": "Comments", "url": "https://www.bytebase.com/blog/30saas-services-behind-startup", "title": "SaaS services behind a startup", "date_modified": "2022-09-26T12:47:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/chatwoot/chatwoot", "title": "Show HN: Open-Source Intercom with Help Center", "date_modified": "2022-09-26T07:09:48.000Z" }, { "content_html": "Comments", "url": "https://fusectore.dev/2022/09/25/github-actions-pitfalls.html", "title": "GitHub Actions Pitfalls", "date_modified": "2022-09-25T10:32:04.000Z" }, { "content_html": "Comments", "url": "https://apisix.apache.org/blog/2022/08/12/arm-performance-google-aws-azure-with-apisix/", "title": "GCP, AWS, and Azure ARM-based server performance comparison", "date_modified": "2022-09-24T23:44:21.000Z" }, { "content_html": "Comments", "url": "https://kaleidawave.github.io/posts/introducing-ezno/", "title": "Ezno: a new TypeScript compiler", "date_modified": "2022-09-23T11:38:22.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/", "title": "Pingora, the proxy that connects Cloudflare to the Internet", "date_modified": "2022-09-14T13:11:50.000Z" }, { "content_html": "Comments", "url": "https://earthly.dev/blog/bazel-build/", "title": "When to use Bazel?", "date_modified": "2022-09-13T18:46:21.000Z" }, { "content_html": "Comments", "url": "https://growing-products.paralect.com/a-development-process-startup-founders-should-use-to-ship-features-weirdly-fast", "title": "A development process startup founders should use to ship features weirdly fast", "date_modified": "2022-09-12T21:08:33.000Z" }, { "content_html": "Comments", "url": "https://xeiaso.net/talks/how-my-website-works", "title": "Serving a high-performance blog website from memory only, using Rust", "date_modified": "2022-09-12T12:40:41.000Z" }, { "content_html": "Comments", "url": "https://grahamc.com/blog/nix-and-layered-docker-images", "title": "Optimising Docker Layers for Better Caching with Nix (2018)", "date_modified": "2022-09-09T15:26:50.000Z" }, { "content_html": "Comments", "url": "https://www.relate.so/blog/six-applications-y-combinator/", "title": "Our five failed YC applications and one successful one", "date_modified": "2022-09-08T20:00:54.000Z" }, { "content_html": "Comments", "url": "https://blog.quarkslab.com/defeating-ebpf-uprobe-monitoring.html", "title": "Defeating eBPF Uprobe Monitoring", "date_modified": "2022-09-08T16:44:32.000Z" }, { "content_html": "Comments", "url": "https://www.graplsecurity.com/post/attacking-firecracker", "title": "Attacking Firecracker: AWS' MicroVM Monitor Written in Rust", "date_modified": "2022-09-08T16:20:36.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/fresh-1.1", "title": "Fresh 1.1 – automatic JSX, plugins, DevTools, and more", "date_modified": "2022-09-08T13:28:29.000Z" }, { "content_html": "Comments", "url": "https://dl.acm.org/doi/10.1145/3132747.3132763", "title": "My VM is lighter (and safer) than your container (2017)", "date_modified": "2022-09-08T12:26:46.000Z" }, { "content_html": "Comments", "url": "https://www.ycombinator.com/blog/meet-the-yc-summer-2022-batch/", "title": "The YC Summer 2022 Batch", "date_modified": "2022-09-07T15:01:12.000Z" }, { "content_html": "Comments", "url": "https://determinate.systems/posts/introducing-riff", "title": "Riff, automatically provide external dependencies for Rust projects", "date_modified": "2022-09-06T17:05:06.000Z" }, { "content_html": "Comments", "url": "https://www.drawanything.app/", "title": "Show HN: Draw Anything – A Simple Stable Diffusion Playground", "date_modified": "2022-09-05T17:16:31.000Z" }, { "content_html": "Comments", "url": "https://determinate.systems/posts/we-want-to-make-nix-better", "title": "We want to make Nix better", "date_modified": "2022-09-02T17:03:29.000Z" }, { "content_html": "Comments", "url": "https://drewdevault.com/2022/09/02/2022-09-02-In-praise-of-qemu.html", "title": "In Praise of QEMU", "date_modified": "2022-09-02T10:08:03.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/cloudflares-abuse-policies-and-approach/", "title": "Cloudflare's abuse policies & approach", "date_modified": "2022-08-31T13:13:09.000Z" }, { "content_html": "Comments", "url": "https://thenewstack.io/javascript-hydration-is-a-workaround-not-a-solution/", "title": "JavaScript hydration is a workaround, not a solution", "date_modified": "2022-08-30T14:34:36.000Z" }, { "content_html": "Comments", "url": "https://acorn.io/", "title": "Acorn: A simple application deployment framework for Kubernetes", "date_modified": "2022-08-27T23:08:31.000Z" }, { "content_html": "Comments", "url": "https://devblogs.microsoft.com/dotnet/announcing-builtin-container-support-for-the-dotnet-sdk/", "title": "Built-in container support for the .NET SDK", "date_modified": "2022-08-27T20:53:15.000Z" }, { "content_html": "

Comments

", "url": "https://kenneth.io/post/developer-experience-infrastructure-dxi", "title": "Developer Experience Infrastructure", "date_modified": "2022-08-27T14:40:47.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=32619199", "title": "Ask HN: Should I checkmate my employer?", "date_modified": "2022-08-27T14:09:16.000Z" }, { "content_html": "Comments", "url": "https://blog.chainguard.dev/a-toolbox-for-a-secure-software-supply-chain/", "title": "A toolbox for a secure software supply chain", "date_modified": "2022-08-26T07:40:36.000Z" }, { "content_html": "Comments", "url": "https://github.com/jetpack-io/devbox", "title": "Devbox: Instant, easy, and predictable shells and containers", "date_modified": "2022-08-25T22:35:06.000Z" }, { "content_html": "Comments", "url": "https://github.com/jetpack-io/devbox", "title": "Show HN: Devbox – Easy, predictable shells and containers", "date_modified": "2022-08-25T22:35:06.000Z" }, { "content_html": "Comments", "url": "https://github.blog/changelog/2022-08-23-ssh-commit-verification-now-supported/", "title": "SSH commit verification now supported", "date_modified": "2022-08-23T17:07:11.000Z" }, { "content_html": "Comments", "url": "https://crawlee.dev/", "title": "Show HN: Crawlee – The web scraping and browser automation library for Node.js", "date_modified": "2022-08-23T06:25:39.000Z" }, { "content_html": "Comments", "url": "https://tomassetti.me/parsing-sql/", "title": "Parsing SQL", "date_modified": "2022-08-23T02:58:29.000Z" }, { "content_html": "Comments", "url": "https://fundingfyre.com/", "title": "Show HN: Largest collection of pitch deck, videos and memos to fund my education", "date_modified": "2022-08-22T16:32:27.000Z" }, { "content_html": "Comments", "url": "https://www.matteason.co.uk/scotbeats/", "title": "Ambient Scotrail Beats – Relax to Scottish train announcements over low-fi beats", "date_modified": "2022-08-20T21:56:23.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/introducing-the-planetscale-serverless-driver-for-javascript", "title": "The PlanetScale serverless driver for JavaScript", "date_modified": "2022-08-18T16:13:03.000Z" }, { "content_html": "Comments", "url": "https://electronicmaterialsoffice.com/", "title": "Show HN: I spent a year designing a low profile, minimal mechanical keyboard", "date_modified": "2022-08-18T09:24:13.000Z" }, { "content_html": "Comments", "url": "https://electronicmaterialsoffice.com/", "title": "I spent a year designing a low profile, minimal mechanical keyboard", "date_modified": "2022-08-18T09:24:13.000Z" }, { "content_html": "Comments", "url": "https://nixpacks.com/docs/getting-started", "title": "Nixpacks takes a source directory and produces an OCI compliant image", "date_modified": "2022-08-17T20:43:26.000Z" }, { "content_html": "

Comments

", "url": "https://metalbear.co/blog/hooking-go-from-rust-hitchhikers-guide-to-the-go-laxy/", "title": "Hooking Go from Rust - Hitchhiker’s Guide to the Go-laxy", "date_modified": "2022-08-17T16:25:12.000Z" }, { "content_html": "

Comments

", "url": "https://mental-reverb.com/blog.php?id=37", "title": "When docker images stop being portable", "date_modified": "2022-08-16T21:38:45.000Z" }, { "content_html": "Comments", "url": "https://gittup.org/tup/", "title": "Tup – an instrumenting file-based build system", "date_modified": "2022-08-15T23:44:41.000Z" }, { "content_html": "Comments", "url": "https://github.com/drand/tlock", "title": "Timelock Encryption made possible and easy to use", "date_modified": "2022-08-15T20:54:55.000Z" }, { "content_html": "Comments", "url": "https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency", "title": "How Discord supercharges network disks for extreme low latency", "date_modified": "2022-08-15T19:24:21.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/changes", "title": "Big Changes Ahead for Deno", "date_modified": "2022-08-15T12:06:13.000Z" }, { "content_html": "Comments", "url": "https://neon.tech/blog/paxos/", "title": "Why use Paxos instead of Raft?", "date_modified": "2022-08-15T10:32:17.000Z" }, { "content_html": "Comments", "url": "https://microfounder.com/5k", "title": "Solo developer startups to $5k MRR", "date_modified": "2022-08-15T09:11:03.000Z" }, { "content_html": "Comments", "url": "https://blog.najaryan.net/posts/partial-protobuf-encoding/", "title": "Faster Protocol Buffers", "date_modified": "2022-08-14T06:44:22.000Z" }, { "content_html": "Comments", "url": "https://dragonflydb.io/blog/2022/06/23/cache_design/", "title": "DragonFlydb: Cache Design", "date_modified": "2022-08-13T09:47:51.000Z" }, { "content_html": "Comments", "url": "https://github.com/containers/krunvm", "title": "Krunvm – Create MicroVMs from OCI Images", "date_modified": "2022-08-13T08:40:14.000Z" }, { "content_html": "Comments", "url": "https://tonsky.me/blog/datascript-2/", "title": "Ideas for DataScript 2", "date_modified": "2022-08-13T07:19:07.000Z" }, { "content_html": "

Comments

", "url": "https://lwn.net/Articles/902585/", "title": "Direct host system calls from KVM", "date_modified": "2022-08-12T22:22:10.000Z" }, { "content_html": "Comments", "url": "https://www.plural.sh/blog/kubernetes-statefulsets-are-broken/", "title": "Kubernetes Statefulsets Are Broken", "date_modified": "2022-08-12T14:31:34.000Z" }, { "content_html": "Comments", "url": "https://github.com/0xricksanchez/like-dbg", "title": "Fully Dockerized Linux kernel debugging environment", "date_modified": "2022-08-11T10:42:10.000Z" }, { "content_html": "Comments", "url": "https://github.com/containers/podman/releases/tag/v4.2.0", "title": "Podman 4.2.0", "date_modified": "2022-08-11T05:06:34.000Z" }, { "content_html": "Comments", "url": "https://www.usenix.org/conference/atc20/presentation/rebello", "title": "Can Applications Recover from Fsync Failures?", "date_modified": "2022-08-10T18:10:51.000Z" }, { "content_html": "

Comments

", "url": "https://sluongng.hashnode.dev/bazel-in-ci-part-0-the-developer-experience-funnel", "title": "Developer Experience Funnel", "date_modified": "2022-08-08T17:06:24.000Z" }, { "content_html": "Comments", "url": "http://_.0xffff.me/dynamodb2022.html", "title": "Some notes on DynamoDB 2022 paper", "date_modified": "2022-08-05T20:24:48.000Z" }, { "content_html": "Comments", "url": "https://john-millikin.com/stateless-kubernetes-overlay-networks-with-ipv6", "title": "Stateless Kubernetes overlay networks with IPv6", "date_modified": "2022-08-05T10:53:34.000Z" }, { "content_html": "Comments", "url": "https://github.com/docker-slim", "title": "Minify your container by up to 30x to be more secure (free and open source)", "date_modified": "2022-08-03T17:42:44.000Z" }, { "content_html": "Comments", "url": "https://www.svix.com/blog/guaranteeing-webhook-ordering/", "title": "You Can't Guarantee Webhook Ordering", "date_modified": "2022-08-03T16:17:08.000Z" }, { "content_html": "Comments", "url": "https://github.com/modus-continens/modus", "title": "Modus: A language for building Docker/OCI container images", "date_modified": "2022-08-03T14:16:45.000Z" }, { "content_html": "

Startups are often filled with ambitious ideas, growth-hungry team members, and excellent positioning. But most share one common problem: they lack funding. In fact, nearly 30 percent of startups fail due to inadequate funding. This problem isn't isolated to startups. Eighty-two percent of business closures are directly tied to cash flow struggles. Luckily, there are a wide variety of resources startups can leverage to get funding during the early stages.

One way to get started is to raise seed funding.

Understanding seed funding

Startups looking for funding rarely have positive net cash flow, and many only have a few founders and a great plan. Only 2 in 5 startups are profitable at any point. And most startups struggle to find profits early. According to some sources, it takes around 2 to 3 years for the average startup to start generating profits. Many founders do not have the personal wealth to bootstrap their startups for years.

Because traditional funding levers (e.g., private equity firms, investment banks, hedge funds, etc.) aren't willing to fund unproven businesses, startups often look for a special type of funding called \"seed\" funding. As the name suggests, investors provide a seed (funding) in the hopes that the startup nurtures that seed money into a healthy tree (a successful startup). In return, the startup provides equity (usually between 5 to 20 percent) or convertible debt.

So, where do you find seed funding?

The \"who's who\" of seed funding

Generally, seed funding comes from one of the following sources:

Each of these funding levers provides unique benefits. Friends and family are often the easiest way to get quick funding. But at the risk of damaged relationships. Angels, incubators, and accelerators all provide a wealth of intangible benefits, but they can be challenging to get into, depending on the startup's business plan. And venture capital usually works best when your startup already has a well-established framework.

Resources to help you raise capital for your startup

Understanding who provides seed funding isn't the same as actually getting seed funding. Many startups exist outside of massive cities, and not all founders are well-connected. Thankfully, a significant amount of funding is now done digitally. There are many websites and services aimed at providing startups with the resources and experience they need to grow a successful company. These include:

Angel-finding websites: There are a wide variety of angel funding websites that directly connect founders with angels across the globe. Some of the most popular include:

Incubators: There are thousands of incubators across the United States alone, but some stand out more than others. The most successful incubators include:

Crowdfunding websites: Crowdfunding is a great way to quickly raise funds for your business. You essentially get funded by many different anonymous people in return for equity. While GoFundMe and Kickstarter are great for product-based startups, most new startups go through seed-based crowdfunding rounds on websites like: 

Loans: You always have the option to take out loans and microloans. Small businesses can use SBA loans to get some quick capital, and there are a wide variety of unique loan options for startups. For example, solutions like Pipe give you up-front capital in return for recurring revenue (which may be ideal for some SaaS companies looking for hands-off funding). You can find loan options at your bank or through a variety of different websites.

In addition to these funding options, there are many resources for startups that aren't related to capital. Startup events, networking websites, forums, social media, and a wide variety of websites exist to help founders connect to other founders and business leaders. Growing a successful business takes more than money. You should always look to build connections and gain insights from industry leaders along the way.

Partner with us

DigitalOcean provides founders a variety of resources through Hatch, our global startup program. Sign up to learn how we can help you build and grow your startup through intelligent infrastructure solutions.

", "url": "https://www.digitalocean.com/blog/learn-the-basics-of-seed-funding", "title": "How to raise seed capital for your startup | DigitalOcean", "date_modified": "2022-08-03T12:22:55.000Z" }, { "content_html": "Comments", "url": "https://github.com/ValdikSS/nat-traversal-github-actions-openvpn-wireguard", "title": "OpenVPN & WireGuard server at GitHub Actions: representative NAT traversal case", "date_modified": "2022-08-03T08:23:57.000Z" }, { "content_html": "Comments", "url": "http://dallery.gallery/wp-content/uploads/2022/07/The-DALL%C2%B7E-2-prompt-book-v1.02.pdf", "title": "DALL·E 2 prompt book [pdf]", "date_modified": "2022-08-02T18:14:03.000Z" }, { "content_html": "Comments", "url": "https://stanislas.blog/2021/08/firecracker/", "title": "Using Firecracker and Go to run short-lived, untrusted code execution jobs", "date_modified": "2022-08-02T12:01:53.000Z" }, { "content_html": "

Comments

", "url": "https://stanislas.blog/2021/08/firecracker/", "title": "Using Firecracker and Go to run short-lived, untrusted code execution jobs", "date_modified": "2022-08-01T14:59:06.000Z" }, { "content_html": "

Comments

", "url": "https://sluongng.hashnode.dev/bazel-caching-explained-pt-3-repository-cache", "title": "Bazel Repository Cache", "date_modified": "2022-08-01T13:45:32.000Z" }, { "content_html": "

Comments

", "url": "https://lwn.net/SubscriberLink/902049/374614a66c0367f3/", "title": "Docker and the OCI container ecosystem", "date_modified": "2022-08-01T01:17:45.000Z" }, { "content_html": "Comments", "url": "https://github.com/losfair/mvsqlite", "title": "Show HN: Distributed SQLite on FoundationDB", "date_modified": "2022-07-28T19:49:05.000Z" }, { "content_html": "Comments", "url": "https://chunk.run", "title": "Show HN: Chunk – Code sandbox for back-end devs", "date_modified": "2022-07-28T17:49:59.000Z" }, { "content_html": "Comments", "url": "https://www.edgedb.com/blog/edgedb-2-0", "title": "EdgeDB 2.0", "date_modified": "2022-07-28T17:12:36.000Z" }, { "content_html": "Comments", "url": "https://deepfence.io/aya-your-trusty-ebpf-companion/", "title": "Aya: your tRusty eBPF companion", "date_modified": "2022-07-28T16:51:28.000Z" }, { "content_html": "Comments", "url": "https://matt-rickard.com/10-years-and-s3-isnt-getting-cheaper/", "title": "S3 isn't getting cheaper", "date_modified": "2022-07-28T16:17:51.000Z" }, { "content_html": "

Hello! The other day we talked about what happened when you press a key in your terminal.

\n\n

As a followup, I thought it might be fun to implement a program that’s like a\ntiny ssh server, but without the security. You can find it on github here, and I’ll explain how it works in this blog post.

\n\n

the goal: “ssh” to a remote computer

\n\n

Our goal is to be able to login to a remote computer and run commands, like you\ndo with SSH or telnet.

\n\n

The biggest difference between this program and SSH is that there’s literally\nno security (not even a password) – anyone who can make a TCP connection to\nthe server can get a shell and run commands.

\n\n

Obviously this is not a useful program in real life, but our goal is to learn a\nlittle more about how terminals works, not to write a useful program.

\n\n

(I will run a version of it on the public internet for the next week though,\nyou can see how to connect to it at the end of this blog post)

\n\n

let’s start with the server!

\n\n

We’re also going to write a client, but the server is the interesting part, so\nlet’s start there. We’re going to write a server that listens on a TCP port (I\npicked 7777) and creates remote terminals for any client that connects to it to\nuse.

\n\n

When the server receives a new connection it needs to:

\n\n
    \n
  1. create a pseudoterminal for the client to use
    2. \n
  2. start a bash shell process for the client to use
    3. \n
  3. connect bash to the pseudoterminal
    4. \n
  4. continuously copy information back and forth between the TCP connection and\nthe pseudoterminal
    5. \n
\n\n

I just said the word “pseudoterminal” a lot, so let’s talk about what that\nmeans.

\n\n

what’s a pseudoterminal?

\n\n

Okay, what the heck is a pseudoterminal?

\n\n

A pseudoterminal is a lot like a bidirectional pipe or a socket – you have two\nends, and they can both send and receive information. You can read more about\nthe information being sent and received in what happens if you press a key in your terminal

\n\n

Basically the idea is that on one end, we have a TCP connection, and on the\nother end, we have a bash shell. So we need to hook one part of the\npseudoterminal up to the TCP connection and the other end to bash.

\n\n

The two parts of the pseudoterminal are called:

\n\n\n\n

Once they’re conected, we can communicate with bash over our TCP connection\nand we’ll have a remote shell!

\n\n

why do we need this “pseudoterminal” thing anyway?

\n\n

You might be wondering – Julia, if a pseudoterminal is kind of like a socket,\nwhy can’t we just set our bash shell’s stdout / stderr / stdin to the TCP\nsocket?

\n\n

And you can! We could write a TCP connection handler like this that does exactly that, it’s not a lot of code (server-notty.go).

\n\n
\nfunc handle(conn net.Conn) {\n\ttty, _ := conn.(*net.TCPConn).File()\n\t// start bash with tcp connection as stdin/stdout/stderr\n\tcmd := exec.Command(\"bash\")\n\tcmd.Stdin = tty\n\tcmd.Stdout = tty\n\tcmd.Stderr = tty\n\tcmd.Start()\n}\n\n
\n\n

It even kind of works – if we connect to it with nc localhost 7778, we can\nrun commands and look at their output.

\n\n

But there are a few problems. I’m not going to list all of them, just two.

\n\n

problem 1: Ctrl + C doesn’t work

\n\n

The way Ctrl + C works in a remote login session is

\n\n\n\n

If the “terminal” is just a TCP connection, this doesn’t work, because when you\nsend 0x04 to a TCP connection, Linux won’t magically send SIGINT to any\nprocess.

\n\n

problem 2: top doesn’t work

\n\n

When I try to run top in this shell, I get the error message top: failed tty get. If we strace it, we see this system call:

\n\n
ioctl(2, TCGETS, 0x7ffec4e68d60)        = -1 ENOTTY (Inappropriate ioctl for device)\n
\n\n

So top is running an ioctl on its output file descriptor (2) to get some\ninformation about the terminal. But Linux is like “hey, this isn’t a terminal!”\nand returns an error.

\n\n

There are a bunch of other things that go wrong, but hopefully at this point\nyou’re convinced that we actually need to set bash’s stdout/stderr to be a\nterminal, not some other thing like a socket.

\n\n

So let’s start looking at the server code and see what creating a\npseudoterminal actually looks like.

\n\n

step 1: create a pseudoterminal

\n\n

Here’s some Go code to create a pseudoterminal on Linux. This is copied from github.com/creack/pty,\nbut I removed some of the error handling to make the logic a bit easier to follow:

\n\n
pty, _ := os.OpenFile(\"/dev/ptmx\", os.O_RDWR, 0)\nsname := ptsname(p)\nunlockpt(p)\ntty, _ := os.OpenFile(sname, os.O_RDWR|syscall.O_NOCTTY, 0)\n
\n\n

In English, what we’re doing is:

\n\n\n\n

What do those ptsname and unlockpt functions do? They just make some\nioctl system calls to the Linux kernel. All of the communication with the\nLinux kernel about terminals seems to be through various ioctl system calls.

\n\n

Here’s the code, it’s pretty short: (again, I just copied it from creack/pty)

\n\n
func ptsname(f *os.File) string {\n\tvar n uint32\n\tioctl(f.Fd(), syscall.TIOCGPTN, uintptr(unsafe.Pointer(&n)))\n\treturn \"/dev/pts/\" + strconv.Itoa(int(n))\n}\n\nfunc unlockpt(f *os.File) {\n\tvar u int32\n\t// use TIOCSPTLCK with a pointer to zero to clear the lock\n\tioctl(f.Fd(), syscall.TIOCSPTLCK, uintptr(unsafe.Pointer(&u)))\n}\n
\n\n

step 2: hook the pseudoterminal up to bash

\n\n

The next thing we have to do is connect the pseudoterminal to bash. Luckily,\nthat’s really easy – here’s the Go code for it! We just need to start a new\nprocess and set the stdin, stdout, and stderr to tty.

\n\n
cmd := exec.Command(\"bash\")\ncmd.Stdin = tty\ncmd.Stdout = tty\ncmd.Stderr = tty\ncmd.SysProcAttr = &syscall.SysProcAttr{\n  Setsid: true,\n}\ncmd.Start()\n
\n\n

Easy! Though – why do we need this Setsid: true thing, you might ask? Well,\nI tried commenting out that code to see what went wrong. It turns out that what\ngoes wrong is – Ctrl + C doesn’t work anymore!

\n\n

Setsid: true creates a new session for the new bash process. But why does\nthat make Ctrl + C work? How does Linux know which process to send SIGINT\nto when you press Ctrl + C, and what does that have to do with sessions?

\n\n

how does Linux know which process to send Ctrl + C to?

\n\n

I found this pretty confusing, so I reached for my favourite book for learning\nabout this kind of thing: the linux programming interface, specifically chapter 34 on process groups\nand sessions.

\n\n

That chapter contains a few key facts: (#3, #4, and #5 are direct quotes from the book)

\n\n
    \n
  1. Every process has a session id and a process group id (which may or may not be the same as its PID)
    2. \n
  2. A session is made up of multiple process groups
    3. \n
  3. All of the processes in a session share a single controlling terminal.
    4. \n
  4. A terminal may be the controlling terminal of at most one session.
    5. \n
  5. At any point in time, one of the process groups in a session is the\nforeground process group for the terminal, and the others are background\nprocess groups.
    6. \n
  6. When you press Ctrl+C in a terminal, SIGINT gets sent to all the processes in the foreground process group
    7. \n
\n\n

What’s a process group? Well, my understanding is that:

\n\n\n\n

I didn’t know most of this (I had no idea processes had a session ID!) so this\nwas kind of a lot to absorb. I tried to draw a sketchy ASCII art diagram of the\nsituation

\n\n
(maybe)  terminal --- session --- process group --- process\n                               |                 |- process\n                               |                 |- process\n                               |- process group \n                               |\n                               |- process group \n
\n\n

So when we press Ctrl+C in a terminal, here’s what I think happens:

\n\n\n\n

If we don’t create a new session for our new bash process, our new pseudoterminal\nactually won’t have any session associated with it, so nothing happens when\nwe press Ctrl+C. But if we do create a new session, then the new\npseudoterminal will have the new session associated with it.

\n\n

how to get a list of all your sessions

\n\n

As a quick aside, if you want to get a list of all the sessions on your Linux\nmachine, grouped by session, you can run:

\n\n
$ ps -eo user,pid,pgid,sess,cmd | sort -k3\n
\n\n

This includes the PID, process group ID, and session ID. As an example of the output, here are the two processes in the pipeline:

\n\n
bork       58080   58080   57922 ps -eo user,pid,pgid,sess,cmd\nbork       58081   58080   57922 sort -k3\n
\n\n

You can see that they share the same process group ID and session ID, but of\ncourse they have different PIDs.

\n\n

That was kind of a lot but that’s all we’re going to say about sessions and\nprocess groups in this post. Let’s keep going!

\n\n

step 3: set the window size

\n\n

We need to tell the terminal how big to be!

\n\n

Again, I just copied this from creack/pty. I decided to hardcode the size to 80x24.

\n\n
Setsize(tty, &Winsize{\n\t\tCols: 80,\n\t\tRows: 24,\n\t})\n
\n\n

Like with getting the terminal’s pts filename and unlocking it, setting the\nsize is just one ioctl system call:

\n\n
func Setsize(t *os.File, ws *Winsize) {\n\tioctl(t.Fd(), syscall.TIOCSWINSZ, uintptr(unsafe.Pointer(ws)))\n}\n
\n\n

Pretty simple! We could do something smarter and get the real window size, but\nI’m too lazy.

\n\n

step 4: copy information between the TCP connection and the pseudoterminal

\n\n

As a reminder, our rough steps to set up this remote login server were:

\n\n
    \n
  1. create a pseudoterminal for the client to use
    2. \n
  2. start a bash shell process
    3. \n
  3. connect bash to the pseudoterminal
    4. \n
  4. continuously copy information back and forth between the TCP connection and\nthe pseudoterminal
    5. \n
\n\n

We’ve done 1, 2, and 3, now we just need to ferry information between the TCP\nconnection and the pseudoterminal.

\n\n

There are two io.Copy calls, one to copy the input from the tcp connection, and one to copy the output to the TCP connection. Here’s what the code looks like:

\n\n
\tgo func() {\n\t\t\tio.Copy(pty, conn)\n\t}()\n  io.Copy(conn, pty)\n
\n\n

The first one is in a goroutine just so they can both run in parallel.

\n\n

Pretty simple!

\n\n

step 5: exit when we’re done

\n\n

I also added a little bit of code to close the TCP connection when the command exits

\n\n
go func() {\n  cmd.Wait()\n  conn.Close()\n}()\n\n
\n\n

And that’s it for the server! You can see all of the Go code here: server.go.

\n\n

next: write a client

\n\n

Next, we have to write a client. This is a lot easier than the server because we don’t need to do quite as much terminal setup. There are just 3 steps:

\n\n
    \n
  1. Put the terminal into raw mode
    2. \n
  2. copy stdin/stdout to the TCP connection
    3. \n
  3. reset the terminal
    4. \n
\n\n

client step 1: put the terminal into “raw” mode

\n\n

We need to put the client terminal into “raw” mode so that every time you press\na key, it gets sent to the TCP connection immediately. If we don’t do this,\neverything will only get sent when you press enter.

\n\n

“Raw mode” isn’t actually a single thing, it’s a bunch of flags that you want\nto turn off. There’s a good tutorial explaining all the flags we have to turn\noff called Entering raw mode.

\n\n

Like everything else with terminals, this requires ioctl system calls. In\nthis case we get the terminal’s current settings, modify them, and save the old\nsettings so that we can restore them later.

\n\n

I figured out how to do this in Go by going to grep.app and typing in\nsyscall.TCSETS to find some other Go code that was doing the same thing.

\n\n
func MakeRaw(fd uintptr) syscall.Termios {\n\t// from https://github.com/getlantern/lantern/blob/devel/archive/src/golang.org/x/crypto/ssh/terminal/util.go\n\tvar oldState syscall.Termios\n\tioctl(fd, syscall.TCGETS, uintptr(unsafe.Pointer(&oldState)))\n\n\tnewState := oldState\n\tnewState.Iflag &^= syscall.ISTRIP | syscall.INLCR | syscall.ICRNL | syscall.IGNCR | syscall.IXON | syscall.IXOFF\n\tnewState.Lflag &^= syscall.ECHO | syscall.ICANON | syscall.ISIG\n\tioctl(fd, syscall.TCSETS, uintptr(unsafe.Pointer(&newState)))\n\treturn oldState\n}\n
\n\n

client step 2: copy stdin/stdout to the TCP connection

\n\n

This is exactly like what we did with the server. It’s very little code:

\n\n
go func() {\n\t\tio.Copy(conn, os.Stdin)\n\t}()\n\tio.Copy(os.Stdout, conn)\n
\n\n

client step 3: restore the terminal’s state

\n\n

We can put the terminal back into the mode it started in like this (another ioctl!):

\n\n
func Restore(fd uintptr, oldState syscall.Termios) {\n\tioctl(fd, syscall.TCSETS, uintptr(unsafe.Pointer(&oldState)))\n}\n
\n\n

we did it!

\n\n

We have written a tiny remote login server that lets anyone log in! Hooray!

\n\n

Obviously this has zero security so I’m not going to talk about that aspect.

\n\n

it’s running on the public internet! you can try it out!

\n\n

For the next week or so I’m going to run a demo of this on the internet at\ntetris.jvns.ca. It runs tetris instead of a shell because I wanted to avoid\nabuse, but if you want to try it with a shell you can run it on your own\ncomputer :).

\n\n

If you want to try it out, you can use netcat as a client instead of the\ncustom Go client program we wrote, because copying information to/from a TCP\nconnection is what netcat does. Here’s how:

\n\n
stty raw -echo && nc tetris.jvns.ca 7777 && stty sane\n
\n\n

This will let you play a terminal tetris game called tint.

\n\n

You can also use the client.go program and run go run client.go tetris.jvns.ca 7777.

\n\n

this is not a good protocol

\n\n

This protocol where we just copy bytes from the TCP connection to the terminal\nand nothing else is not good because it doesn’t allow us to send over\ninformation information like the terminal or the actual window size of the\nterminal.

\n\n

I thought about implementing telnet’s protocol so that we could use telnet as a\nclient, but I didn’t feel like figuring out how telnet works so I didn’t. (the\nserver 30% works with telnet as is, but a lot of things are broken, I don’t\nquite know why, and I didn’t feel like figuring it out)

\n\n

it’ll mess up your terminal a bit

\n\n

As a warning: using this server to play tetris will probably mess up your\nterminal a bit because it sets the window size to 80x24. To fix that I just\nclosed the terminal tab after running that command.

\n\n

If we wanted to fix this for real, we’d need to restore the window size after\nwe’re done, but then we’d need a slightly more real protocol than “just\nblindly copy bytes back and forth with TCP” and I didn’t feel like doing that.

\n\n

Also it sometimes takes a second to disconnect after the program exits for some\nreason, I’m not sure why that is.

\n\n

other tiny projects

\n\n

That’s all! There are a couple of other similar toy implementations of programs\nI’ve written here:

\n\n", "url": "https://jvns.ca/blog/2022/07/28/toy-remote-login-server/", "title": "A toy remote login server", "date_modified": "2022-07-28T08:00:28.000Z" }, { "content_html": "Comments", "url": "https://www.thenile.dev/blog/multi-tenant-rls", "title": "Shipping Multi-Tenant SaaS Using Postgres Row-Level Security", "date_modified": "2022-07-26T18:16:22.000Z" }, { "content_html": "Comments", "url": "https://matt-rickard.com/non-obvious-docker-uses/", "title": "Non-Obvious Docker Uses", "date_modified": "2022-07-24T14:44:10.000Z" }, { "content_html": "Comments", "url": "https://dr-emann.github.io/squashfs/squashfs.html", "title": "Squashfs binary format", "date_modified": "2022-07-24T12:39:26.000Z" }, { "content_html": "Comments", "url": "https://www.nongnu.org/lzip/xz_inadequate.html", "title": "Xz format considered inadequate for long-term archiving (2016)", "date_modified": "2022-07-24T04:52:02.000Z" }, { "content_html": "Comments", "url": "https://www.bbc.com/news/science-environment-62225696", "title": "The audacious PR plot that seeded doubt about climate change", "date_modified": "2022-07-23T23:00:05.000Z" }, { "content_html": "Comments", "url": "https://vercel.com/blog/build-output-api", "title": "Vercel Build Output API: Infrastructure-as-Filesystem", "date_modified": "2022-07-22T14:53:23.000Z" }, { "content_html": "Comments", "url": "https://github.com/supabase/pg_jsonschema", "title": "Show HN: Pg_jsonschema – A Postgres extension for JSON validation", "date_modified": "2022-07-21T14:31:20.000Z" }, { "content_html": "Comments", "url": "https://www.hetzner.com/news/07-22-rx-line/", "title": "Hetzner to Offer First Arm-Based Dedicated Servers in Europe", "date_modified": "2022-07-20T11:37:04.000Z" }, { "content_html": "Comments", "url": "https://brandur.org/soft-deletion", "title": "Soft Deletion Probably Isn't Worth It", "date_modified": "2022-07-19T18:35:39.000Z" }, { "content_html": "
\"Using
\"Using

Cloudflare has been using Kafka in production since 2014. We have come a long way since then, and currently run 14 distinct Kafka clusters, across multiple data centers, with roughly 330 nodes. Between them, over a trillion messages have been processed over the last eight years.

Cloudflare uses Kafka to decouple microservices and communicate the creation, change or deletion of various resources via a common data format in a fault-tolerant manner. This decoupling is one of many factors that enables Cloudflare engineering teams to work on multiple features and products concurrently.

We learnt a lot about Kafka on the way to one trillion messages, and built some interesting internal tools to ease adoption that will be explored in this blog post. The focus in this blog post is on inter-application communication use cases alone and not logging (we have other Kafka clusters that power the dashboards where customers view statistics that handle more than one trillion messages each day). I am an engineer on the Application Services team and our team has a charter to provide tools/services to product teams, so they can focus on their core competency which is delivering value to our customers.

In this blog I’d like to recount some of our experiences in the hope that it helps other engineering teams who are on a similar journey of adopting Kafka widely.

Tooling

One of our Kafka clusters is creatively named Messagebus. It is the most general purpose cluster we run, and was created to:

To make it as easy to use as possible and to encourage adoption, the Application Services team created two internal projects. The first is unimaginatively named Messagebus-Client. Messagebus-Client is a Go library that wraps the fantastic Shopify Sarama library with an opinionated set of configuration options and the ability to manage the rotation of mTLS certificates.

\"Using

The success of this project is also somewhat its downfall. By providing a ready-to-go Kafka client, we ensured teams got up and running quickly, but we also abstracted some core concepts of Kafka a little too much, meaning that small unassuming configuration changes could have a big impact.

One such example led to partition skew (a large portion of messages being directed towards a single partition, meaning we were not processing messages in real time; see the chart below). One drawback of Kafka is you can only have one consumer per partition, so when incidents do occur, you can’t trivially scale your way to faster throughput.

That also means before your service hits production it is wise to do some back of the napkin math to figure out what throughput might look like, otherwise you will need to add partitions later. We have since amended our library to make events like the below less likely.

\"Using

The reception for the Messagebus-Client has been largely positive. We spent time as a team to understand what the predominant use cases were, and took the concept one step further to build out what we call the connector framework.

Connectors

The connector framework is based on Kafka-connectors and allows our engineers to easily spin up a service that can read from a system of record and push it somewhere else (such as Kafka, or even Cloudflare’s own Quicksilver). To make this as easy as possible, we use Cookiecutter templating to allow engineers to enter a few parameters into a CLI and in return receive a ready to deploy service.

\"Using

We provide the ability to configure data pipelines via environment variables. For simple use cases, we provide the functionality out of the box. However, extending the readers, writers and transformations is as simple as satisfying an interface and “registering” the new entry.

For example, adding the environment variables:

READER=kafka\nTRANSFORMATIONS=topic_router:topic1,topic2|pf_edge\nWRITER=quicksilver\n
\n

will:

Connectors come readily baked with basic metrics and alerts, so teams know they can move to production quickly but with confidence.

Below is a diagram of how one team used our connector framework to read from the Messagebus cluster and write to various other systems. This is orchestrated by a system the Application Service team runs called Communication Preferences Service (CPS). Whenever a user opts in/out of marketing emails or changes their language preferences on cloudflare.com, they are calling CPS which ensures those settings are reflected in all the relevant systems.

\"Using

Strict Schemas

Alongside the Messagebus-Client library, we also provide a repo called Messagebus Schema. This is a schema registry for all message types that will be sent over our Messagebus cluster. For message format, we use protobuf and have been very happy with that decision. Previously, our team had used JSON for some of our kafka schemas, but we found it much harder to enforce forward and backwards compatibility, as well as message sizes being substantially larger than the protobuf equivalent. Protobuf provides strict message schemas (including type safety), the forward and backwards compatibility we desired, the ability to generate code in multiple languages as well as the files being very human-readable.

We encourage heavy commentary before approving a merge. Once merged, we use prototool to do breaking change detection, enforce some stylistic rules and to generate code for various languages (at time of writing it's just Go and Rust, but it is trivial to add more).

\"Using
An example Protobuf message in our schema

Furthermore, in Messagebus Schema we store a mapping of proto messages to a team, alongside that team’s chat room in our internal communication tool. This allows us to escalate issues to the correct team easily when necessary.

One important decision we made for the Messagebus cluster is to only allow one proto message per topic. This is configured in Messagebus Schema and enforced by the Messagebus-Client. This was a good decision to enable easy adoption, but it has led to numerous topics existing. When you consider that for each topic we create, we add numerous partitions and replicate them with a replication factor of at least three for resilience, there is a lot of potential to optimize compute for our lower throughput topics.

Observability

Making it easy for teams to observe Kafka is essential for our decoupled engineering model to be successful. We therefore have automated metrics and alert creation wherever we can to ensure that all the engineering teams have a wealth of information available to them to respond to any issues that arise in a timely manner.

We use Salt to manage our infrastructure configuration and follow a Gitops style model, where our repo holds the source of truth for the state of our infrastructure. To add a new Kafka topic, our engineers make a pull request into this repo and add a couple of lines of YAML. Upon merge, the topic and an alert for high lag (where lag is defined as the difference in time between the last committed offset being read and the last produced offset being produced) will be created. Other alerts can (and should) be created, but this is left to the discretion of application teams. The reason we automatically generate alerts for high lag is that this simple alert is a great proxy for catching a high amount of issues including:

For metrics, we use Prometheus and display them with Grafana. For each new topic created, we automatically provide a view into production rate, consumption rate and partition skew by producer/consumer. If an engineering team is called out, within the alert message is a link to this Grafana view.

\"Using

In our Messagebus-Client, we expose some metrics automatically and users get the ability to extend them further. The metrics we expose by default are:

For producers:

For consumer:

Some teams use these for alerting on a significant change in throughput, others use them to alert if no messages are produced/consumed in a given time frame.

A Practical Example

As well as providing the Messagebus framework, the Application Services team looks for common concerns within Engineering and looks to solve them in a scalable, extensible way which means other engineering teams can utilize the system and not have to build their own (thus meaning we are not building lots of disparate systems that are only slightly different).

One example is the Alert Notification System (ANS). ANS is the backend service for the “Notifications” tab in the Cloudflare dashboard. You may have noticed over the past 12 months that new alert and policy types have been made available to customers very regularly. This is because we have made it very easy for other teams to do this. The approach is:

That’s it! The producer team now has a means for customers to configure granular alerting policies for their new alert that includes being able to dispatch them via Slack, Google Chat or a custom webhook, PagerDuty or email (by both API and dashboard). Retrying and dead letter messages are managed for them, and a whole host of metrics are made available, all by making some very small changes.

\"Using

What’s Next?

Usage of Kafka (and our Messagebus tools) is only going to increase at Cloudflare as we continue to grow, and as a team we are committed to making the tooling around Messagebus easy to use, customizable where necessary and (perhaps most importantly) easy to observe. We regularly take feedback from other engineers to help improve the Messagebus-Client (we are on the fifth version now) and are currently experimenting with abstracting the intricacies of Kafka away completely and allowing teams to use gRPC to stream messages to Kafka. Blog post on the success/failure of this to follow!

If you're interested in building scalable services and solving interesting technical problems, we are hiring engineers on our team in Austin, and Remote US.

", "url": "https://blog.cloudflare.com/using-apache-kafka-to-process-1-trillion-messages/", "title": "Using Apache Kafka to process 1 trillion inter-service messages", "date_modified": "2022-07-19T13:00:00.000Z" }, { "content_html": "Comments", "url": "https://about.sourcegraph.com/blog/ex-googler-guide-dev-tools", "title": "An ex-Googler's guide to dev tools", "date_modified": "2022-07-18T02:50:46.000Z" }, { "content_html": "Comments", "url": "https://basicappleguy.com/basicappleblog/999month", "title": "$9.99/Month", "date_modified": "2022-07-17T18:28:50.000Z" }, { "content_html": "Comments", "url": "https://earthly.dev/blog/programming-language-improvements/", "title": "The slow march of progress in programming language tooling", "date_modified": "2022-07-17T16:32:12.000Z" }, { "content_html": "Comments", "url": "https://deno.com/blog/2022-07-13-outage-post-mortem", "title": "Deno’s July 13th incident update", "date_modified": "2022-07-16T08:07:26.000Z" }, { "content_html": "Comments", "url": "https://medium.com/@codervinod/i-deleted-78-of-my-redis-container-and-it-still-works-df8310a3a007", "title": "I deleted 78% of my Redis container and it still works", "date_modified": "2022-07-16T07:57:09.000Z" }, { "content_html": "Comments", "url": "https://devblogs.microsoft.com/engineering-at-microsoft/microsoft-open-sources-salus-software-bill-of-materials-sbom-generation-tool/", "title": "Microsoft open sources Salus software bill of materials (SBOM) generation tool", "date_modified": "2022-07-15T05:07:18.000Z" }, { "content_html": "Comments", "url": "https://justine.lol/pledge/", "title": "Show HN: Porting OpenBSD Pledge() to Linux", "date_modified": "2022-07-14T14:52:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/Permify/permify", "title": "Show HN: Permify – Open-source authorization service based on Google Zanzibar", "date_modified": "2022-07-14T14:38:14.000Z" }, { "content_html": "Comments", "url": "https://github.com/Qovery/Replibyte", "title": "Replibyte – Seed your database with real data", "date_modified": "2022-07-10T18:39:08.000Z" }, { "content_html": "Comments", "url": "https://jvns.ca/blog/2022/07/09/monitoring-small-web-services/", "title": "Monitoring tiny web services", "date_modified": "2022-07-09T17:29:02.000Z" }, { "content_html": "Comments", "url": "https://changelog.com/podcast/496", "title": "Oxide Builds Servers", "date_modified": "2022-07-09T17:15:58.000Z" }, { "content_html": "Comments", "url": "https://code.visualstudio.com/blogs/2022/07/07/vscode-server", "title": "The Visual Studio Code Server", "date_modified": "2022-07-08T09:41:32.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/soc2-the-screenshots-will-continue-until-security-improves/", "title": "SOC2: The Screenshots Will Continue Until Security Improves", "date_modified": "2022-07-07T19:14:19.000Z" }, { "content_html": "
\"Announcing
\"Announcing

Today, we are announcing experimental support for WASI (the WebAssembly System Interface) on Cloudflare Workers and support within wrangler2 to make it a joy to work with. We continue to be incredibly excited about the entire WebAssembly ecosystem and are eager to adopt the standards as they are developed.

A Quick Primer on WebAssembly

So what is WASI anyway? To understand WASI, and why we’re excited about it, it’s worth a quick recap of WebAssembly, and the ecosystem around it.

WebAssembly promised us a future in which code written in compiled languages could be compiled to a common binary format and run in a secure sandbox, at near native speeds. While WebAssembly was designed with the browser in mind, the model rapidly extended to server-side platforms such as Cloudflare Workers (which has supported WebAssembly since 2017).

WebAssembly was originally designed to run alongside Javascript, and requires developers to interface directly with Javascript in order to access the world outside the sandbox. To put it another way, WebAssembly does not provide any standard interface for I/O tasks such as interacting with files, accessing the network, or reading the system clock. This means if you want to respond to an event from the outside world, it's up to the developer to handle that event in JavaScript, and directly call functions exported from the WebAssembly module. Similarly, if you want to perform I/O from within WebAssembly, you need to implement that logic in Javascript and import it into the WebAssembly module.

Custom toolchains such as Emscripten or libraries such as wasm-bindgen have emerged to make this easier, but they are language specific and add a tremendous amount of complexity and bloat. We've even built our own library, workers-rs, using wasm-bindgen that attempts to make writing applications in Rust feel native within a Worker – but this has proven not only difficult to maintain, but requires developers to write code that is Workers specific, and is not portable outside the Workers ecosystem.

We need more.

The WebAssembly System Interface (WASI)

WASI aims to provide a standard interface that any language compiling to WebAssembly can target. You can read the original post by Lin Clark here, which gives an excellent introduction – code cartoons and all. In a nutshell, Lin describes WebAssembly as an assembly language for a 'conceptual machine', whereas WASI is a systems interface for a ‘conceptual operating system.’

This standardization of the system interface has paved the way for existing toolchains to cross-compile existing codebases to the wasm32-wasi target. A tremendous amount of progress has already been made, specifically within Clang/LLVM via the wasi-sdk and Rust toolchains. These toolchains leverage a version of Libc, which provides POSIX standard API calls, that is built on top of WASI 'system calls.' There are even basic implementations in more fringe toolchains such as TinyGo and SwiftWasm.

Practically speaking, this means that you can now write applications that not only interoperate with any WebAssembly runtime implementing the standard, but also any POSIX compliant system! This means the exact same 'Hello World!' that runs on your local Linux/Mac/Windows WSL machine.

Show me the code

WASI sounds great, but does it actually make my life easier? You tell us. Let’s run through an example of how this would work in practice.

First, let’s generate a basic Rust “Hello, world!” application, compile, and run it.

$ cargo new hello_world\n$ cd ./hello_world\n$ cargo build --release\n   Compiling hello_world v0.1.0 (/Users/benyule/hello_world)\n    Finished release [optimized] target(s) in 0.28s\n$ ./target/release/hello_world\nHello, world!\n
\n

It doesn’t get much simpler than this. You’ll notice we only define a main() function followed by a println to stdout.

fn main() {\n    println!(\"Hello, world!\");\n}\n
\n

Now, let’s take the exact same program and compile against the wasm32-wasi target, and run it in an ‘off the shelf’ wasm runtime such as Wasmtime.

$ cargo build --target wasm32-wasi --release\n$ wasmtime target/wasm32-wasi/release/hello_world.wasm\n\nHello, world!\n
\n

Neat! The same code compiles and runs in multiple POSIX environments.

Finally, let’s take the binary we just generated for Wasmtime, but instead publish it to Workers using Wrangler2.

$ npx wrangler@wasm dev target/wasm32-wasi/release/hello_world.wasm\n$ curl http://localhost:8787/\n\nHello, world!\n
\n

Unsurprisingly, it works! The same code is compatible in multiple POSIX environments and the same binary is compatible across multiple WASM runtimes.

Running your CLI apps in the cloud

The attentive reader may notice that we played a small trick with the HTTP request made via cURL. In this example, we actually stream stdin and stdout to/from the Worker using the HTTP request and response body respectively. This pattern enables some really interesting use cases, specifically, programs designed to run on the command line can be deployed as 'services' to the cloud.

‘Hexyl’ is an example that works completely out of the box. Here, we ‘cat’ a binary file on our local machine and ‘pipe’ the output to curl, which will then POST that output to our service and stream the result back. Following the steps we used to compile our 'Hello World!', we can compile hexyl.

$ git clone git@github.com:sharkdp/hexyl.git\n$ cd ./hexyl\n$ cargo build --target wasm32-wasi --release\n
\n

And without further modification we were able to take a real-world program and create something we can now run or deploy. Again, let's tell wrangler2 to preview hexyl, but this time give it some input.

$ npx wrangler@wasm dev target/wasm32-wasi/release/hexyl.wasm\n$ echo \"Hello, world\\!\" | curl -X POST --data-binary @- http://localhost:8787\n\n┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐\n│00000000│ 48 65 6c 6c 6f 20 77 6f ┊ 72 6c 64 21 0a          │Hello wo┊rld!_   │\n└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘\n\n
\n

Give it a try yourself by hitting https://hexyl.examples.workers.dev.

echo \"Hello world\\!\" | curl https://hexyl.examples.workers.dev/ -X POST --data-binary @- --output -\n
\n

A more useful example, but requires a bit more work, would be to deploy a utility such as swc (swc.rs), to the cloud and use it as an on demand JavaScript/TypeScript transpilation service. Here, we have a few extra steps to ensure that the compiled output is as small as possible, but it otherwise runs out-of-the-box. Those steps are detailed in github.com/zebp/wasi-example-swc, but for now let’s gloss over that and interact with the hosted example.

$ echo \"const x = (x, y) => x * y;\" | curl -X POST --data-binary @- https://swc-wasi.examples.workers.dev/ --output -\n\nvar x=function(a,b){return a*b}\n
\n

Finally, we can also do the same with C/C++, but requires a little more lifting to get our Makefile right. Here we show an example of compiling zstd and uploading it as a streaming compression service.

github.com/zebp/wasi-example-zstd

$ echo \"Hello world\\!\" | curl https://zstd.examples.workers.dev/ -s -X POST --data-binary @- | file -\n
\n

What if I want to use WASI from within a JavaScript Worker?

Wrangler can make it really easy to deploy code without having to worry about the Workers ecosystem, but in some cases you may actually want to invoke your WASI based WASM module from Javascript. This can be achieved with the following simple boilerplate. An updated README will be kept at github.com/cloudflare/workers-wasi.

import { WASI } from \"@cloudflare/workers-wasi\";\nimport demoWasm from \"./demo.wasm\";\n\nexport default {\n  async fetch(request, _env, ctx) {\n    // Creates a TransformStream we can use to pipe our stdout to our response body.\n    const stdout = new TransformStream();\n    const wasi = new WASI({\n      args: [],\n      stdin: request.body,\n      stdout: stdout.writable,\n    });\n\n    // Instantiate our WASM with our demo module and our configured WASI import.\n    const instance = new WebAssembly.Instance(demoWasm, {\n      wasi_snapshot_preview1: wasi.wasiImport,\n    });\n\n    // Keep our worker alive until the WASM has finished executing.\n    ctx.waitUntil(wasi.start(instance));\n\n    // Finally, let's reply with the WASM's output.\n    return new Response(stdout.readable);\n  },\n};\n
\n

Now with our JavaScript boilerplate and wasm, we can easily deploy our worker with Wrangler’s WASM feature.

$ npx wrangler publish\nTotal Upload: 473.89 KiB / gzip: 163.79 KiB\nUploaded wasi-javascript (2.75 sec)\nPublished wasi-javascript (0.30 sec)\n  wasi-javascript.zeb.workers.dev\n
\n

Back to the future

For those of you who have been around for the better part of the past couple of decades, you may notice this looks very similar to RFC3875, better known as CGI (The Common Gateway Interface). While our example here certainly does not conform to the specification, you can imagine how this can be extended to turn the stdin of a basic 'command line' application into a full-blown http handler.

We are thrilled to learn where developers take this from here. Share what you build with us on Discord or Twitter!

...
\nWe protect entire corporate networks, help customers build Internet-scale applications efficiently, accelerate any website or Internet application, ward off DDoS attacks, keep hackers at bay, and can help you on your journey to Zero Trust.

\n

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.To learn more about our mission to help build a better Internet, start here. If you’re looking for a new career direction, check out our open positions.

", "url": "https://blog.cloudflare.com/announcing-wasi-on-workers/", "title": "Announcing support for WASI on Cloudflare Workers", "date_modified": "2022-07-07T16:09:43.000Z" }, { "content_html": "Comments", "url": "https://www.smashingmagazine.com/2022/07/designing-better-pricing-page/", "title": "Designing a Better Pricing Page", "date_modified": "2022-07-07T10:27:50.000Z" }, { "content_html": "Comments", "url": "https://bun.sh/?launch", "title": "Bun: Fast JavaScript runtime, transpiler, and NPM client written in Zig", "date_modified": "2022-07-05T20:41:53.000Z" }, { "content_html": "Comments", "url": "https://github.com/citronneur/pamspy", "title": "Show HN: Credentials dumper for Linux using eBPF", "date_modified": "2022-07-05T14:44:55.000Z" }, { "content_html": "Comments", "url": "https://www.jeremybrown.tech/8-kubernetes-is-a-red-flag-signalling-premature-optimisation/", "title": "Kubernetes is a red flag signalling premature optimisation", "date_modified": "2022-07-04T07:27:03.000Z" }, { "content_html": "Comments", "url": "https://www.karltarvas.com/2020/10/25/macos-app-sandboxing-via-sandbox-exec.html", "title": "macOS: App sandboxing via sandbox-exec", "date_modified": "2022-07-04T03:32:37.000Z" }, { "content_html": "Comments", "url": "https://www.karltarvas.com/2020/10/25/macos-app-sandboxing-via-sandbox-exec.html", "title": "macOS: App sandboxing via sandbox-exec (2020)", "date_modified": "2022-07-04T03:32:37.000Z" }, { "content_html": "Comments", "url": "https://matt-rickard.com/the-end-of-ci/", "title": "The End of CI", "date_modified": "2022-07-03T05:14:39.000Z" }, { "content_html": "Comments", "url": "https://michael.stapelberg.ch/posts/2022-07-02-rsync-how-does-it-work/", "title": "rsync, article 3: How does rsync work?", "date_modified": "2022-07-02T12:41:23.000Z" }, { "content_html": "Comments", "url": "https://github.com/GauravDawra/Beast", "title": "Show HN: Beast – The Build System", "date_modified": "2022-07-02T11:04:12.000Z" }, { "content_html": "Comments", "url": "https://www.theatlantic.com/technology/archive/2022/07/us-sunscreen-ingredients-outdated-technology-better-eu-asia/661433/", "title": "You're Not Allowed to Have the Best Sunscreens in the World", "date_modified": "2022-07-01T23:19:01.000Z" }, { "content_html": "
\"Optimizing
\"Optimizing

Here at Cloudflare we're constantly working on improving our service. Our engineers are looking at hundreds of parameters of our traffic, making sure that we get better all the time.

One of the core numbers we keep a close eye on is HTTP request latency, which is important for many of our products. We regard latency spikes as bugs to be fixed. One example is the 2017 story of \"Why does one NGINX worker take all the load?\", where we optimized our TCP Accept queues to improve overall latency of TCP sockets waiting for accept().

Performance tuning is a holistic endeavor, and we monitor and continuously improve a range of other performance metrics as well, including throughput. Sometimes, tradeoffs have to be made. Such a case occurred in 2015, when a latency spike was discovered in our processing of HTTP requests. The solution at the time was to set tcp_rmem to 4 MiB, which minimizes the amount of time the kernel spends on TCP collapse processing. It was this collapse processing that was causing the latency spikes. Later in this post we discuss TCP collapse processing in more detail.

The tradeoff is that using a low value for tcp_rmem limits TCP throughput over high latency links. The following graph shows the maximum throughput as a function of network latency for a window size of 2 MiB. Note that the 2 MiB corresponds to a tcp_rmem value of 4 MiB due to the tcp_adv_win_scale setting in effect at the time.

\"Optimizing

For the Cloudflare products then in existence, this was not a major problem, as connections terminate and content is served from nearby servers due to our BGP anycast routing.

Since then, we have added new products, such as Magic WAN, WARP, Spectrum, Gateway, and others. These represent new types of use cases and traffic flows.

For example, imagine you're a typical Magic WAN customer. You have connected all of your worldwide offices together using the Cloudflare global network. While Time to First Byte still matters, Magic WAN office-to-office traffic also needs good throughput. For example, a lot of traffic over these corporate connections will be file sharing using protocols such as SMB. These are elephant flows over long fat networks. Throughput is the metric every eyeball watches as they are downloading files.

We need to continue to provide world-class low latency while simultaneously providing high throughput over high-latency connections.

Before we begin, let’s introduce the players in our game.

TCP receive window is the maximum number of unacknowledged user payload bytes the sender should transmit (bytes-in-flight) at any point in time. The size of the receive window can and does go up and down during the course of a TCP session. It is a mechanism whereby the receiver can tell the sender to stop sending if the sent packets cannot be successfully received because the receive buffers are full. It is this receive window that often limits throughput over high-latency networks.

net.ipv4.tcp_adv_win_scale is a (non-intuitive) number used to account for the overhead needed by Linux to process packets. The receive window is specified in terms of user payload bytes. Linux needs additional memory beyond that to track other data associated with packets it is processing.

The value of the receive window changes during the lifetime of a TCP session, depending on a number of factors. The maximum value that the receive window can be is limited by the amount of free memory available in the receive buffer, according to this table:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
tcp_adv_win_scaleTCP window size
415/16 * available memory in receive buffer
3⅞ * available memory in receive buffer
2¾ * available memory in receive buffer
1½ * available memory in receive buffer
0available memory in receive buffer
-1½ * available memory in receive buffer
-2¼ * available memory in receive buffer
-3⅛ * available memory in receive buffer
\n

We can intuitively (and correctly) understand that the amount of available memory in the receive buffer is the difference between the used memory and the maximum limit. But what is the maximum size a receive buffer can be? The answer is sk_rcvbuf.

sk_rcvbuf is a per-socket field that specifies the maximum amount of memory that a receive buffer can allocate. This can be set programmatically with the socket option SO_RCVBUF. This can sometimes be useful to do, for localhost TCP sessions, for example, but in general the use of SO_RCVBUF is not recommended.

So how is sk_rcvbuf set? The most appropriate value for that depends on the latency of the TCP session and other factors. This makes it difficult for L7 applications to know how to set these values correctly, as they will be different for every TCP session. The solution to this problem is Linux autotuning.

Linux autotuning

Linux autotuning is logic in the Linux kernel that adjusts the buffer size limits and the receive window based on actual packet processing. It takes into consideration a number of things including TCP session RTT, L7 read rates, and the amount of available host memory.

Autotuning can sometimes seem mysterious, but it is actually fairly straightforward.

The central idea is that Linux can track the rate at which the local application is reading data off of the receive queue. It also knows the session RTT. Because Linux knows these things, it can automatically increase the buffers and receive window until it reaches the point at which the application layer or network bottleneck links are the constraint on throughput (and not host buffer settings). At the same time, autotuning prevents slow local readers from having excessively large receive queues. The way autotuning does that is by limiting the receive window and its corresponding receive buffer to an appropriate size for each socket.

The values set by autotuning can be seen via the Linux “ss” command from the iproute package (e.g. “ss -tmi”).  The relevant output fields from that command are:

Recv-Q is the number of user payload bytes not yet read by the local application.

rcv_ssthresh is the window clamp, a.k.a. the maximum receive window size. This value is not known to the sender. The sender receives only the current window size, via the TCP header field. A closely-related field in the kernel, tp->window_clamp, is the maximum window size allowable based on the amount of available memory. rcv_ssthresh is the receiver-side slow-start threshold value.

skmem_r is the actual amount of memory that is allocated, which includes not only user payload (Recv-Q) but also additional memory needed by Linux to process the packet (packet metadata). This is known within the kernel as sk_rmem_alloc.

Note that there are other buffers associated with a socket, so skmem_r does not represent the total memory that a socket might have allocated. Those other buffers are not involved in the issues presented in this post.

skmem_rb is the maximum amount of memory that could be allocated by the socket for the receive buffer. This is higher than rcv_ssthresh to account for memory needed for packet processing that is not packet data. Autotuning can increase this value (up to tcp_rmem max) based on how fast the L7 application is able to read data from the socket and the RTT of the session. This is known within the kernel as sk_rcvbuf.

rcv_space is the high water mark of the rate of the local application reading from the receive buffer during any RTT. This is used internally within the kernel to adjust sk_rcvbuf.

Earlier we mentioned a setting called tcp_rmem. net.ipv4.tcp_rmem consists of three values, but in this document we are always referring to the third value (except where noted). It is a global setting that specifies the maximum amount of memory that any TCP receive buffer can allocate, i.e. the maximum permissible value that autotuning can use for sk_rcvbuf. This is essentially just a failsafe for autotuning, and under normal circumstances should play only a minor role in TCP memory management.

It’s worth mentioning that receive buffer memory is not preallocated. Memory is allocated based on actual packets arriving and sitting in the receive queue. It’s also important to realize that filling up a receive queue is not one of the criteria that autotuning uses to increase sk_rcvbuf. Indeed, preventing this type of excessive buffering (bufferbloat) is one of the benefits of autotuning.

What’s the problem?

The problem is that we must have a large TCP receive window for high BDP sessions. This is directly at odds with the latency spike problem mentioned above.

Something has to give. The laws of physics (speed of light in glass, etc.) dictate that we must use large window sizes. There is no way to get around that. So we are forced to solve the latency spikes differently.

A brief recap of the latency spike problem

Sometimes a TCP session will fill up its receive buffers. When that happens, the Linux kernel will attempt to reduce the amount of memory the receive queue is using by performing what amounts to a “defragmentation” of memory. This is called collapsing the queue. Collapsing the queue takes time, which is what drives up HTTP request latency.

We do not want to spend time collapsing TCP queues.

Why do receive queues fill up to the point where they hit the maximum memory limit? The usual situation is when the local application starts out reading data from the receive queue at one rate (triggering autotuning to raise the max receive window), followed by the local application slowing down its reading from the receive queue. This is valid behavior, and we need to handle it correctly.

Selecting sysctl values

Before exploring solutions, let’s first decide what we need as the maximum TCP window size.

As we have seen above in the discussion about BDP, the window size is determined based upon the RTT and desired throughput of the connection.

Because Linux autotuning will adjust correctly for sessions with lower RTTs and bottleneck links with lower throughput, all we need to be concerned about are the maximums.

For latency, we have chosen 300 ms as the maximum expected latency, as that is the measured latency between our Zurich and Sydney facilities. It seems reasonable enough as a worst-case latency under normal circumstances.

For throughput, although we have very fast and modern hardware on the Cloudflare global network, we don’t expect a single TCP session to saturate the hardware. We have arbitrarily chosen 3500 mbps as the highest supported throughput for our highest latency TCP sessions.

The calculation for those numbers results in a BDP of 131MB, which we round to the more aesthetic value of 128 MiB.

Recall that allocation of TCP memory includes metadata overhead in addition to packet data. The ratio of actual amount of memory allocated to user payload size varies, depending on NIC driver settings, packet size, and other factors. For full-sized packets on some of our hardware, we have measured average allocations up to 3 times the packet data size. In order to reduce the frequency of TCP collapse on our servers, we set tcp_adv_win_scale to -2. From the table above, we know that the max window size will be ¼ of the max buffer space.

We end up with the following sysctl values:

net.ipv4.tcp_rmem = 8192 262144 536870912\nnet.ipv4.tcp_wmem = 4096 16384 536870912\nnet.ipv4.tcp_adv_win_scale = -2\n
\n

A tcp_rmem of 512MiB and tcp_adv_win_scale of -2 results in a maximum window size that autotuning can set of 128 MiB, our desired value.

Disabling TCP collapse

Patient: Doctor, it hurts when we collapse the TCP receive queue.

Doctor: Then don’t do that!

Generally speaking, when a packet arrives at a buffer when the buffer is full, the packet gets dropped. In the case of these receive buffers, Linux tries to “save the packet” when the buffer is full by collapsing the receive queue. Frequently this is successful, but it is not guaranteed to be, and it takes time.

There are no problems created by immediately just dropping the packet instead of trying to save it. The receive queue is full anyway, so the local receiver application still has data to read. The sender’s congestion control will notice the drop and/or ZeroWindow and will respond appropriately. Everything will continue working as designed.

At present, there is no setting provided by Linux to disable the TCP collapse. We developed an in-house patch to the kernel to disable the TCP collapse logic.

Kernel patch – Attempt #1

The kernel patch for our first attempt was straightforward. At the top of tcp_try_rmem_schedule(), if the memory allocation fails, we simply return (after pred_flag = 0 and tcp_sack_reset()), thus completely skipping the tcp_collapse and related logic.

It didn’t work.

Although we eliminated the latency spikes while using large buffer limits, we did not observe the throughput we expected.

One of the realizations we made as we investigated the situation was that standard network benchmarking tools such as iperf3 and similar do not expose the problem we are trying to solve. iperf3 does not fill the receive queue. Linux autotuning does not open the TCP window large enough. Autotuning is working perfectly for our well-behaved benchmarking program.

We need application-layer software that is slightly less well-behaved, one that exercises the autotuning logic under test. So we wrote one.

A new benchmarking tool

Anomalies were seen during our “Attempt #1” that negatively impacted throughput. The anomalies were seen only under certain specific conditions, and we realized we needed a better benchmarking tool to detect and measure the performance impact of those anomalies.

This tool has turned into an invaluable resource during the development of this patch and raised confidence in our solution.

It consists of two Python programs. The reader opens a TCP session to the daemon, at which point the daemon starts sending user payload as fast as it can, and never stops sending.

The reader, on the other hand, starts and stops reading in a way to open up the TCP receive window wide open and then repeatedly causes the buffers to fill up completely. More specifically, the reader implemented this logic:

    \n
  1. reads as fast as it can, for five seconds\n\n
    2. \n
  2. calculates 5% of the high watermark of the bytes reader during any previous one second
    3. \n
  3. for each second of the next 15 seconds:\n\n
    4. \n
  4. steps 1-3 are repeated in a loop three times, so the entire run is 60 seconds
    5. \n
\n

This has the effect of highlighting any issues in the handling of packets when the buffers repeatedly hit the limit.

Revisiting default Linux behavior

Taking a step back, let’s look at the default Linux behavior. The following is kernel v5.15.16.

\"Optimizing

The Linux kernel is effective at freeing up space in order to make room for incoming packets when the receive buffer memory limit is hit. As documented previously, the cost for saving these packets (i.e. not dropping them) is latency.

However, the latency spikes, in milliseconds, for tcp_try_rmem_schedule(), are:

tcp_rmem 170 MiB, tcp_adv_win_scale +2 (170p2):

@ms:\n[0]       27093 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n[1]           0 |\n[2, 4)        0 |\n[4, 8)        0 |\n[8, 16)       0 |\n[16, 32)      0 |\n[32, 64)     16 |\n
\n

tcp_rmem 146 MiB, tcp_adv_win_scale +3 (146p3):

@ms:\n(..., 16)  25984 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n[16, 20)       0 |\n[20, 24)       0 |\n[24, 28)       0 |\n[28, 32)       0 |\n[32, 36)       0 |\n[36, 40)       0 |\n[40, 44)       1 |\n[44, 48)       6 |\n[48, 52)       6 |\n[52, 56)       3 |\n
\n

tcp_rmem 137 MiB, tcp_adv_win_scale +4 (137p4):

@ms:\n(..., 16)  37222 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n[16, 20)       0 |\n[20, 24)       0 |\n[24, 28)       0 |\n[28, 32)       0 |\n[32, 36)       0 |\n[36, 40)       1 |\n[40, 44)       8 |\n[44, 48)       2 |\n
\n

These are the latency spikes we cannot have on the Cloudflare global network.

Kernel patch – Attempt #2

So the “something” that was not working in Attempt #1 was that the receive queue memory limit was hit early on as the flow was just ramping up (when the values for sk_rmem_alloc and sk_rcvbuf were small, ~800KB). This occurred at about the two second mark for 137p4 test (about 2.25 seconds for 170p2).

In hindsight, we should have noticed that tcp_prune_queue() actually raises sk_rcvbuf when it can. So we modified the patch in response to that, added a guard to allow the collapse to execute when sk_rmem_alloc is less than the threshold value.

net.ipv4.tcp_collapse_max_bytes = 6291456

The next section discusses how we arrived at this value for tcp_collapse_max_bytes.

The patch is available here.

The results with the new patch are as follows:

oscil – 300ms tests

\"Optimizing

oscil – 20ms tests

\"Optimizing

oscil – 0ms tests

\"Optimizing

iperf3 – 300 ms tests

\"Optimizing

iperf3 – 20 ms tests

\"Optimizing

iperf3 – 0ms tests

\"Optimizing

All tests are successful.

Setting tcp_collapse_max_bytes

In order to determine this setting, we need to understand what the biggest queue we can collapse without incurring unacceptable latency.

\"Optimizing
\"Optimizing

Using 6 MiB should result in a maximum latency of no more than 2 ms.

Cloudflare production network results

Current production settings (“Old”)

net.ipv4.tcp_rmem = 8192 2097152 16777216\nnet.ipv4.tcp_wmem = 4096 16384 33554432\nnet.ipv4.tcp_adv_win_scale = -2\nnet.ipv4.tcp_collapse_max_bytes = 0\nnet.ipv4.tcp_notsent_lowat = 4294967295\n
\n

tcp_collapse_max_bytes of 0 means that the custom feature is disabled and that the vanilla kernel logic is used for TCP collapse processing.

New settings under test (“New”)

net.ipv4.tcp_rmem = 8192 262144 536870912\nnet.ipv4.tcp_wmem = 4096 16384 536870912\nnet.ipv4.tcp_adv_win_scale = -2\nnet.ipv4.tcp_collapse_max_bytes = 6291456\nnet.ipv4.tcp_notsent_lowat = 131072\n
\n

The tcp_notsent_lowat setting is discussed in the last section of this post.

The middle value of tcp_rmem was changed as a result of separate work that found that Linux autotuning was setting receive buffers too high for localhost sessions. This updated setting reduces TCP memory usage for those sessions, but does not change anything about the type of TCP sessions that is the focus of this post.

For the following benchmarks, we used non-Cloudflare host machines in Iowa, US, and Melbourne, Australia performing data transfers to the Cloudflare data center in Marseille, France. In Marseille, we have some hosts configured with the existing production settings, and others with the system settings described in this post. Software used is perf3 version 3.9, kernel 5.15.32.

Throughput results

\"Optimizing
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
RTT (ms)Throughput with Current Settings (mbps)Throughput with New Settings (mbps)Increase Factor
Iowa to Marseille121276660024x
Melbourne to Marseille282120380032x
\n

Iowa-Marseille throughput

\"Optimizing

Iowa-Marseille receive window and bytes-in-flight

\"Optimizing

Melbourne-Marseille throughput

\"Optimizing

Melbourne-Marseille receive window and bytes-in-flight

\"Optimizing

Even with the new settings in place, the Melbourne to Marseille performance is limited by the receive window on the Cloudflare host. This means that further adjustments to these settings yield even higher throughput.

Latency results

The Y-axis on these charts are the 99th percentile time for TCP collapse in seconds.

Cloudflare hosts in Marseille running the current production settings

\"Optimizing

Cloudflare hosts in Marseille running the new settings

\"Optimizing

The takeaway in looking at these graphs is that maximum TCP collapse time for the new settings is no worse than with the current production settings. This is the desired result.

Send Buffers

What we have shown so far is that the receiver side seems to be working well, but what about the sender side?

As part of this work, we are setting tcp_wmem max to 512 MiB. For oscillating reader flows, this can cause the send buffer to become quite large. This represents bufferbloat and wasted kernel memory, both things that nobody likes or wants.

Fortunately, there is already a solution: tcp_notsent_lowat. This setting limits the size of unsent bytes in the write queue. More details can be found at https://lwn.net/Articles/560082.

The results are significant:

\"Optimizing

The RTT for these tests was 466ms. Throughput is not negatively affected. Throughput is at full wire speed in all cases (1 Gbps). Memory usage is as reported by /proc/net/sockstat, TCP mem.

Our web servers already set tcp_notsent_lowat to 131072 for its sockets. All other senders are using 4 GiB, the default value. We are changing the sysctl so that 131072 is in effect for all senders running on the server.

Conclusion

The goal of this work is to open the throughput floodgates for high BDP connections while simultaneously ensuring very low HTTP request latency.

We have accomplished that goal.

", "url": "https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/", "title": "Optimizing TCP for high WAN throughput while preserving low latency", "date_modified": "2022-07-01T13:00:01.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31943016", "title": "Ask HN: How do you ensure everyone on the team is heard on Slack?", "date_modified": "2022-07-01T06:17:58.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31930935", "title": "Ask HN: Where and how do you find your early adoptors?", "date_modified": "2022-06-30T11:07:13.000Z" }, { "content_html": "

Comments

", "url": "https://twitter.com/JustJake/status/1542316666666614784", "title": "Nixpacks beat Buildpacks by almost two minutes", "date_modified": "2022-06-30T05:44:22.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2022-06-29-improve-git-monorepo-performance-with-a-file-system-monitor/", "title": "Improve Git monorepo performance with a file system monitor", "date_modified": "2022-06-29T19:20:23.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31914087", "title": "Ask HN: What tools are you a 10/10 on?", "date_modified": "2022-06-28T22:48:04.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31892384", "title": "Ask HN: What is your Kubernetes nightmare?", "date_modified": "2022-06-27T09:39:57.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/elesiuta/picosnitch", "title": "picosnitch: Monitor Linux network traffic per executable using BPF", "date_modified": "2022-06-25T19:41:28.000Z" }, { "content_html": "Comments", "url": "https://github.com/ronomon/pure", "title": "Pure: A static analysis file format checker", "date_modified": "2022-06-25T06:15:06.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31859040", "title": "Ask HN: How to level up your technical writing?", "date_modified": "2022-06-24T08:23:29.000Z" }, { "content_html": "Comments", "url": "https://plausible.io/blog/open-source-saas", "title": "How We built a $1M ARR open source SaaS", "date_modified": "2022-06-24T01:54:50.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/898522/9cf50ee3f96f90c1/", "title": "Whatever happened to SHA-256 support in Git?", "date_modified": "2022-06-23T16:47:45.000Z" }, { "content_html": "Comments", "url": "https://blog.pwego.com/why-users-sometimes-prefer-the-less-straightforward-ux/", "title": "Sometimes users prefer the less straightforward UX", "date_modified": "2022-06-22T23:41:52.000Z" }, { "content_html": "Comments", "url": "https://docs.thenile.dev/blog/infrastructure-saas", "title": "Infrastructure SaaS – a control plane first architecture", "date_modified": "2022-06-22T17:42:49.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/tailscale-ssh/", "title": "Tailscale SSH", "date_modified": "2022-06-22T15:14:32.000Z" }, { "content_html": "Comments", "url": "https://survey.stackoverflow.co/2022/", "title": "Stack Overflow Developer Survey 2022", "date_modified": "2022-06-22T15:05:46.000Z" }, { "content_html": "Comments", "url": "https://webauthn.guide/", "title": "Guide to Web Authentication", "date_modified": "2022-06-22T15:02:08.000Z" }, { "content_html": "Comments", "url": "https://ifuckinghatejira.com/", "title": "I Fucking Hate Jira", "date_modified": "2022-06-20T18:40:57.000Z" }, { "content_html": "Comments", "url": "https://github.com/libfuse/sshfs", "title": "Sshfs Is Orphaned", "date_modified": "2022-06-20T16:45:33.000Z" }, { "content_html": "Comments", "url": "https://tuple.app/blog/sso-should-be-table-stakes", "title": "SSO should be table stakes", "date_modified": "2022-06-20T14:34:51.000Z" }, { "content_html": "Comments", "url": "https://blog.scottlogic.com/2022/06/20/state-of-wasm-2022.html", "title": "The State of WebAssembly 2022", "date_modified": "2022-06-20T09:42:25.000Z" }, { "content_html": "Comments", "url": "https://matt-rickard.com/dont-use-kubernetes-yet/", "title": "Don't use Kubernetes yet", "date_modified": "2022-06-19T00:49:39.000Z" }, { "content_html": "Comments", "url": "https://github.com/hwayne/awesome-cold-showers", "title": "Cold Showers", "date_modified": "2022-06-18T19:34:33.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31782200", "title": "Ask HN: Best Dev Tool pitches of all time?", "date_modified": "2022-06-17T18:18:41.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31782200", "title": "Ask HN: Best dev tool pitches of all time?", "date_modified": "2022-06-17T18:18:41.000Z" }, { "content_html": "Comments", "url": "https://www.cockroachlabs.com/blog/consistency-model/", "title": "CockroachDB's Consistency Model", "date_modified": "2022-06-16T20:30:14.000Z" }, { "content_html": "

Web streams are a standard for streams that is now supported on all major web platforms: web browsers, Node.js, and Deno. (Streams are an abstraction for reading and writing data sequentially in small pieces from all kinds of sources – files, data hosted on servers, etc.)

\n

For example, the global function fetch() (which downloads online resources) asynchronously returns a Response which has a property .body with a web stream.

\n

This blog post covers web streams on Node.js, but most of what we learn applies to all web platforms that support them.

\n\n
[Read rest of post]
", "url": "https://2ality.com/2022/06/web-streams-nodejs.html", "title": "Using web streams on Node.js", "date_modified": "2022-06-16T00:00:00.000Z" }, { "content_html": "Comments", "url": "https://neon.tech/blog/hello-world/", "title": "Select ’Hello, World’: Serverless Postgres Built for the Cloud", "date_modified": "2022-06-15T14:23:52.000Z" }, { "content_html": "

In an interview published today, Bethesda’s creative director Todd Howard revealed the studio’s future plans, explaining that after the Elder Scrolls 6 comes out, the studio’s next game will be Fallout 5, the next main entry in the company’s post-apocalyptic open-world RPG franchise. However, considering how long it…

Read more...

", "url": "https://kotaku.com/fallout-5-v-bethesda-elder-scrolls-6-release-date-starf-1849062025", "title": "Fallout 5 Is Bethesda’s Next Game After Elder Scrolls 6, Will Probably Be Out By 2050", "date_modified": "2022-06-14T22:17:52.000Z" }, { "content_html": "Comments", "url": "https://blog.heroku.com/april-2022-incident-review", "title": "Heroku April 2022 Incident Review", "date_modified": "2022-06-14T17:43:20.000Z" }, { "content_html": "Comments", "url": "https://www.courier.com/blog/the-developers-guide-to-saas-compliance/", "title": "The Developer's Guide to SaaS Compliance", "date_modified": "2022-06-13T17:32:15.000Z" }, { "content_html": "Comments", "url": "https://fresh.deno.dev/", "title": "Fresh – Next-gen web framework", "date_modified": "2022-06-13T01:50:22.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/897435/3f00904f520a1956/", "title": "Vetting the Cargo", "date_modified": "2022-06-13T00:16:48.000Z" }, { "content_html": "Comments", "url": "https://seirdy.one/posts/2022/06/10/cli-best-practices/", "title": "Best Practices for Inclusive CLIs", "date_modified": "2022-06-11T23:01:11.000Z" }, { "content_html": "Comments", "url": "https://blog.sigstore.dev/introducing-gitsign-9fd3f1b682aa", "title": "Gitsign", "date_modified": "2022-06-09T20:36:58.000Z" }, { "content_html": "Comments", "url": "https://dx.tips/the-end-of-localhost", "title": "The End of Localhost", "date_modified": "2022-06-08T16:26:08.000Z" }, { "content_html": "Comments", "url": "https://www.saurabhnanda.in/2022/03/24/dhall-a-gateway-drug-to-haskell/", "title": "Dhall: A Gateway Drug to Haskell", "date_modified": "2022-06-07T18:59:15.000Z" }, { "content_html": "

Comments

", "url": "http://lucasfcosta.com/2022/06/01/ux-patterns-cli-tools.html", "title": "UX patterns for CLI tools", "date_modified": "2022-06-06T08:27:14.000Z" }, { "content_html": "Comments", "url": "https://dock.orion3.space/readme.htm", "title": "Painless Desktop containers for everyday development", "date_modified": "2022-06-04T21:07:48.000Z" }, { "content_html": "Comments", "url": "https://content.fme.de/en/blog/docker-is-dead-podman-an-alternative-tool", "title": "Docker is dead? Podman – an alternative tool?", "date_modified": "2022-06-04T00:22:17.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31576353", "title": "Tell HN: Cloudflare prevents transfer-out of domains, sets to 'pendingdelete'", "date_modified": "2022-05-31T23:53:23.000Z" }, { "content_html": "Comments", "url": "https://somafm.com/", "title": "SomaFM", "date_modified": "2022-05-31T17:08:18.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31568138", "title": "Ask HN: Docker vs simple DLLs?", "date_modified": "2022-05-31T11:02:51.000Z" }, { "content_html": "Comments", "url": "https://github.com/dragonflydb/dragonfly", "title": "Dragonflydb – A modern replacement for Redis and Memcached", "date_modified": "2022-05-30T16:18:35.000Z" }, { "content_html": "Comments", "url": "https://leerob.io/blog/heroku", "title": "The Story of Heroku", "date_modified": "2022-05-30T14:22:15.000Z" }, { "content_html": "

I have found my happy place! Escape Simulator is such a lovely thing, a first-person simulacrum of escape rooms, built in 3D, with realistic physics. It is, as its title suggests, a simulation of attending a real-world escape room, in a way that almost all room-escape video games are not. Apart from when it’s in space.

Read more...

", "url": "https://kotaku.com/escape-simulator-room-pine-studio-review-indie-pc-co-op-1848993246", "title": "This Is As Close To A Real-World Escape Room As You'll Find", "date_modified": "2022-05-30T13:30:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/mgramin/awesome-db-tools", "title": "Everything that makes working with databases easier", "date_modified": "2022-05-29T15:31:20.000Z" }, { "content_html": "Comments", "url": "https://systemfontstack.com", "title": "System Font Stack", "date_modified": "2022-05-28T19:36:14.000Z" }, { "content_html": "Comments", "url": "https://ignaciochiazzo.medium.com/paginating-requests-in-apis-d4883d4c1c4c", "title": "Are all popular APIs moving to Cursor based pagination?", "date_modified": "2022-05-28T15:31:12.000Z" }, { "content_html": "Comments", "url": "https://github.com/richardartoul/tsdb-layer", "title": "FoundationDB Time Series Layer: Millions of writes/s in 2k lines of Go (2019)", "date_modified": "2022-05-28T14:19:16.000Z" }, { "content_html": "Comments", "url": "https://neon.tech", "title": "Neon – Serverless Postgres", "date_modified": "2022-05-28T01:37:10.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31530176", "title": "Ask HN: How can you buy high quality healthcare?", "date_modified": "2022-05-27T14:38:18.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/896267/6bac8e29d614ec66/", "title": "A Linux 5.10 patch has caused user-space regressions", "date_modified": "2022-05-26T16:48:47.000Z" }, { "content_html": "

Comments

", "url": "https://kenkantzer.com/learnings-from-5-years-of-tech-startup-code-audits/", "title": "Learnings from 5 years of tech startup code audits", "date_modified": "2022-05-26T07:37:38.000Z" }, { "content_html": "Comments", "url": "https://www.smashingmagazine.com/2022/05/magical-svg-techniques/", "title": "Magical SVG Techniques", "date_modified": "2022-05-25T09:20:38.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/fly-machines/", "title": "Fly Machines: An API for Fast-Booting VMs", "date_modified": "2022-05-24T19:48:33.000Z" }, { "content_html": "Comments", "url": "https://www.linen.dev/", "title": "Show HN: Linen – Make your Discord community Google-searchable", "date_modified": "2022-05-24T17:19:17.000Z" }, { "content_html": "Comments", "url": "https://flightcontrol.notion.site/Backend-Typescript-Engineer-at-Small-DevTools-Startup-12e52471539a4ab9878514e04c415da6", "title": "Flightcontrol (YC W22) Is hiring a back end TypeScript engineer [Own-cloud PaaS]", "date_modified": "2022-05-22T12:00:26.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31466885", "title": "Ask HN: What developer tools would you like to see?", "date_modified": "2022-05-22T11:06:47.000Z" }, { "content_html": "Comments", "url": "https://github.com/sharkdp/fd", "title": "Fd: A simple, fast and user-friendly alternative to 'find'", "date_modified": "2022-05-20T14:44:17.000Z" }, { "content_html": "Comments", "url": "https://www.helloinbox.email", "title": "Show HN: HelloInbox – Ultimate email deliverability checklist and toolkit", "date_modified": "2022-05-19T07:40:36.000Z" }, { "content_html": "Comments", "url": "https://www.craigkerstiens.com/2022/05/18/unfinished-business-with-postgres/", "title": "Unfinished Business with Postgres", "date_modified": "2022-05-18T17:04:44.000Z" }, { "content_html": "Comments", "url": "https://www.getlago.com/blog/why-billing-systems-are-a-nightmare-for-engineers", "title": "Why billing systems are a nightmare for engineers", "date_modified": "2022-05-18T16:06:44.000Z" }, { "content_html": "Comments", "url": "https://www.datadoghq.com/blog/engineering/introducing-husky/", "title": "Husky, Datadog's Third-Generation Event Store", "date_modified": "2022-05-17T22:04:41.000Z" }, { "content_html": "Comments", "url": "https://blog.hartleybrody.com/thoughts-on-flyio/", "title": "My thoughts about Fly.io (so far) and other newish technology I'm getting into", "date_modified": "2022-05-17T16:59:51.000Z" }, { "content_html": "Comments", "url": "https://www.axiom.co/vercel", "title": "Request logging and web vitals for Vercel apps", "date_modified": "2022-05-16T20:45:08.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/", "title": "Supercharging GitHub Actions with Job Summaries", "date_modified": "2022-05-16T18:28:06.000Z" }, { "content_html": "

Comments

", "url": "https://monzo.com/blog/2022/05/16/how-we-deploy-to-production-over-100-times-a-day/", "title": "How we deploy to production over 100 times a day", "date_modified": "2022-05-16T09:21:44.000Z" }, { "content_html": "Comments", "url": "https://brandur.org/nanoglyphs/033-heroku", "title": "Heroku: Core Impact", "date_modified": "2022-05-15T21:22:13.000Z" }, { "content_html": "Comments", "url": "https://christine.website/blog/fly.io-heroku-replacement", "title": "Fly.io: The reclaimer of Heroku's magic", "date_modified": "2022-05-15T19:56:39.000Z" }, { "content_html": "Comments", "url": "https://matt-rickard.com/why-did-heroku-fail/", "title": "Why Did Heroku Fail?", "date_modified": "2022-05-13T21:22:52.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31357221", "title": "Show HN: I built a service to help companies reduce AWS spend by 50%", "date_modified": "2022-05-12T17:29:02.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/benskuhn/status/1524791658381910023", "title": "Apple Maps location scan spikes WiFi latency every 60 seconds", "date_modified": "2022-05-12T16:56:40.000Z" }, { "content_html": "Comments", "url": "https://www.vladionescu.me/posts/scaling-containers-on-aws-in-2022/", "title": "Benchmarking Container Scaling on AWS", "date_modified": "2022-05-12T10:43:25.000Z" }, { "content_html": "Comments", "url": "https://m-m-pr.com/eurovision-diaries/", "title": "Eurovision Diaries", "date_modified": "2022-05-10T19:24:02.000Z" }, { "content_html": "Comments", "url": "https://arnon.dk/why-you-should-separate-your-billing-from-entitlement/", "title": "Separate Your Billing from Entitlements", "date_modified": "2022-05-10T08:00:21.000Z" }, { "content_html": "

Comments

", "url": "https://ipetkov.dev/blog/introducing-crane/", "title": "Introducing Crane: Composable and Cacheable Builds with Cargo and Nix", "date_modified": "2022-05-06T21:40:51.000Z" }, { "content_html": "Comments", "url": "https://www.madrau.com/support/support/faq_files/ns_Why_isnt_there_any_HiDPI_resolu.html", "title": "Apple Intentionally Disabled HiDPI on M1 Macs to Push 4K Monitors Sales", "date_modified": "2022-05-06T17:39:08.000Z" }, { "content_html": "Comments", "url": "https://startuptrail.engine.is/", "title": "Startup Trail", "date_modified": "2022-05-05T11:06:33.000Z" }, { "content_html": "Comments", "url": "https://kiranrao.ca/2022/05/04/zero-downtime-migrations.html", "title": "Zero downtime migrations", "date_modified": "2022-05-05T04:33:53.000Z" }, { "content_html": "Comments", "url": "https://www.usenix.org/conference/nsdi20/presentation/agache", "title": "Firecracker: Lightweight Virtualization for Serverless Applications (2020)", "date_modified": "2022-05-04T15:16:27.000Z" }, { "content_html": "Comments", "url": "https://wildbit.com/blog/postmark-has-been-acquired-by-activecampaign", "title": "Postmark acquired by marketing firm ActiveCampaign", "date_modified": "2022-05-03T12:07:19.000Z" }, { "content_html": "Comments", "url": "https://wildbit.com/blog/postmark-has-been-acquired-by-activecampaign", "title": "Postmark has been acquired by ActiveCampaign", "date_modified": "2022-05-03T12:07:19.000Z" }, { "content_html": "Comments", "url": "https://plutoapp.xyz/blog/post/designing-a-command-palette/", "title": "Designing a Command Palette", "date_modified": "2022-05-03T04:29:13.000Z" }, { "content_html": "Comments", "url": "https://mail.openjdk.java.net/pipermail/jdk-dev/2022-April/006530.html", "title": "Project Loom preview ships in JDK 19", "date_modified": "2022-05-02T15:58:35.000Z" }, { "content_html": "

Comments

", "url": "http://tla.msr-inria.inria.fr/kuppe/2019conf/06%20-%20William%20Schultz%20-%20Strangeloop%20TLA+%20Conference%202019%20Talk.pdf", "title": "Fixing a MongoDB Replication Protocol Bug with TLA+", "date_modified": "2022-05-02T15:33:45.000Z" }, { "content_html": "Comments", "url": "https://www.brendangregg.com/blog/2022-05-02/brendan-at-intel.html", "title": "Brendan at Intel.com", "date_modified": "2022-05-02T15:12:08.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31230903", "title": "Ask HN: What are your favorite examples of elegant software?", "date_modified": "2022-05-02T02:47:42.000Z" }, { "content_html": "Comments", "url": "https://fasterthanli.me/articles/i-won-free-load-testing", "title": "I won free load testing", "date_modified": "2022-05-02T01:27:25.000Z" }, { "content_html": "Comments", "url": "https://gitbom.dev/resources/whitepaper/", "title": "GitBOM: Enabling universal artifact traceability in software supply chains", "date_modified": "2022-04-30T18:17:18.000Z" }, { "content_html": "Comments", "url": "https://letscooktime.com", "title": "Show HN: Recipes, Not Mommy Blogs", "date_modified": "2022-04-29T04:41:50.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/itchio/wharf", "title": "wharf: A protocol to quickly transfer software builds (reference Go implementation)", "date_modified": "2022-04-29T00:03:55.000Z" }, { "content_html": "Comments", "url": "https://blog.porter.run/why-companies-move-off-heroku/", "title": "Why companies move off Heroku (besides the cost)", "date_modified": "2022-04-27T18:06:03.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31178577", "title": "Ask HN: Do you find it challenging to talk to your users?", "date_modified": "2022-04-27T11:22:55.000Z" }, { "content_html": "Comments", "url": "https://webkit.org/blog/12578/non-interactive-elements-with-the-inert-attribute/", "title": "Non-interactive Elements with the inert attribute", "date_modified": "2022-04-26T20:19:23.000Z" }, { "content_html": "Comments", "url": "https://go.dev/doc/effective_go", "title": "Effective Go", "date_modified": "2022-04-26T14:31:48.000Z" }, { "content_html": "

Comments

", "url": "https://garnix.io/blog/converting-to-flakes", "title": "Nix flakes, and how to convert to them", "date_modified": "2022-04-25T18:34:09.000Z" }, { "content_html": "Comments", "url": "https://www.hyperui.dev/", "title": "Free Open Source Tailwind CSS Components", "date_modified": "2022-04-25T16:04:54.000Z" }, { "content_html": "Comments", "url": "https://m3o.com", "title": "Show HN: M3O – Universal Public API Interface", "date_modified": "2022-04-25T10:09:25.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31152490", "title": "Ask HN: Have you used SQLite as a primary database?", "date_modified": "2022-04-25T10:06:05.000Z" }, { "content_html": "Comments", "url": "https://christine.website/blog/gonads-2022-04-24", "title": "Crimes with Go Generics", "date_modified": "2022-04-25T01:07:06.000Z" }, { "content_html": "Comments", "url": "https://www.textualize.io/", "title": "TUI in webapp design language(CSS) and pattern(check the demo, it’s next level)", "date_modified": "2022-04-24T12:15:36.000Z" }, { "content_html": "Comments", "url": "https://christine.website/talks/nixos-pain-2021-11-10", "title": "How Nix and NixOS Get So Close to Perfect", "date_modified": "2022-04-24T04:49:11.000Z" }, { "content_html": "Comments", "url": "https://ivov.dev/notes/typescript-and-set-theory", "title": "TypeScript and Set Theory", "date_modified": "2022-04-22T00:51:11.000Z" }, { "content_html": "Comments", "url": "https://nixery.dev/", "title": "Nixery – Docker images on the fly with Nix", "date_modified": "2022-04-19T02:13:11.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=31072465", "title": "Ask HN: I burned out but I don't want to let my team down", "date_modified": "2022-04-18T15:59:11.000Z" }, { "content_html": "Comments", "url": "https://danielmangum.com/posts/the-missing-kubernetes-type-system/", "title": "The Missing Kubernetes Type System", "date_modified": "2022-04-17T17:45:48.000Z" }, { "content_html": "Comments", "url": "https://status.heroku.com/incidents/2413", "title": "Heroku Security Incident", "date_modified": "2022-04-16T02:20:10.000Z" }, { "content_html": "Comments", "url": "https://bytebase.com/blog/database-migration-sqlite-to-postgresql", "title": "Migrating from SQLite to PostgreSQL", "date_modified": "2022-04-15T10:42:31.000Z" }, { "content_html": "Comments", "url": "https://read.lukeburgis.com/p/cultural-anorexia", "title": "Cultural Anorexia: The Pursuit of Thicker Desires in a Thinning World", "date_modified": "2022-04-15T08:15:51.000Z" }, { "content_html": "Comments", "url": "https://lexical.dev/", "title": "Facebook open sources Lexical, an extensible text editor library", "date_modified": "2022-04-13T20:22:51.000Z" }, { "content_html": "Comments", "url": "https://www.newyorker.com/culture/rabbit-holes/the-most-popular-chess-streamer-on-twitch", "title": "The most popular chess streamer on Twitch", "date_modified": "2022-04-13T12:27:32.000Z" }, { "content_html": "Comments", "url": "https://grapee.jp/en/199026", "title": "Tokyo’s Manuscript Writing Cafe won't let writers leave until they are finished", "date_modified": "2022-04-13T09:32:35.000Z" }, { "content_html": "Comments", "url": "https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-line-tools/", "title": "A list of new(ish) command line tools – Julia Evans", "date_modified": "2022-04-12T23:08:52.000Z" }, { "content_html": "Comments", "url": "https://www.gov.uk/guidance/protect-domains-that-dont-send-email", "title": "Protect domains that do not send email", "date_modified": "2022-04-11T13:27:10.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30987770", "title": "Ask HN: Editing remote code locally: Best practices?", "date_modified": "2022-04-11T12:28:15.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30976971", "title": "Ask HN: Have you had any real benefits from apps like Headspace, Fabulous, etc.?", "date_modified": "2022-04-10T14:01:43.000Z" }, { "content_html": "Comments", "url": "https://mutagen.io/", "title": "Mutagen – Cloud-based development using your local tools", "date_modified": "2022-04-08T14:24:27.000Z" }, { "content_html": "Comments", "url": "https://missiveapp.com/blog/the-things-we-did-not-do", "title": "Things we did not do while reaching $2M ARR", "date_modified": "2022-04-06T18:27:13.000Z" }, { "content_html": "Comments", "url": "https://squeaky.ai/blog/development/why-we-dont-use-a-staging-environment", "title": "We don’t use a staging environment", "date_modified": "2022-04-03T18:28:42.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/database-for-2022/", "title": "A Database for 2022", "date_modified": "2022-04-01T20:33:11.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/database-for-2022/", "title": "A database for 2022", "date_modified": "2022-04-01T20:33:11.000Z" }, { "content_html": "Comments", "url": "https://github.com/dlvhdr/gh-dash", "title": "Show HN: gh-dash – GitHub CLI dashboard for pull requests and issues", "date_modified": "2022-04-01T18:12:14.000Z" }, { "content_html": "

Comments

", "url": "https://planetscale.com/blog/why-we-chose-nanoids-for-planetscales-api", "title": "Why we chose NanoIDs for PlanetScale's API", "date_modified": "2022-04-01T18:00:44.000Z" }, { "content_html": "Comments", "url": "https://monzo.com/blog/2022/03/31/how-we-secure-monzos-banking-platform/", "title": "How we secure Monzo's banking platform", "date_modified": "2022-04-01T09:36:37.000Z" }, { "content_html": "Comments", "url": "https://amyunger.com/blog/2020/09/10/staff-engineer-at-heroku.html", "title": "How I operated as a Staff engineer at Heroku", "date_modified": "2022-03-31T17:47:00.000Z" }, { "content_html": "Comments", "url": "https://dagger.io/blog/public-launch-announcement", "title": "Dagger: a new way to build CI/CD pipelines", "date_modified": "2022-03-30T16:00:10.000Z" }, { "content_html": "Comments", "url": "https://reactjs.org/blog/2022/03/29/react-v18.html", "title": "React v18.0", "date_modified": "2022-03-29T16:09:50.000Z" }, { "content_html": "Comments", "url": "https://journal.plain.com/posts/2022-02-08-a-magical-aws-serverless-developer-experience/", "title": "A magical AWS serverless developer experience", "date_modified": "2022-03-28T04:05:13.000Z" }, { "content_html": "Comments", "url": "https://rubrikinc.github.io/wachy/", "title": "Show HN: Wachy – A UI for eBPF-based performance debugging", "date_modified": "2022-03-26T15:59:23.000Z" }, { "content_html": "Comments", "url": "https://blog.vamc19.dev/posts/dockerfile-copy-chmod/", "title": "`COPY –chmod` reduced the size of my container image by 35%", "date_modified": "2022-03-26T03:11:58.000Z" }, { "content_html": "Comments", "url": "https://styles.master.co/", "title": "Master Styles – The First Virtual CSS Engine", "date_modified": "2022-03-24T20:10:43.000Z" }, { "content_html": "Comments", "url": "https://github.com/porsager/postgres", "title": "Show HN: Postgres.js – Fastest Full-Featured PostgreSQL Client for Node and Deno", "date_modified": "2022-03-24T19:30:03.000Z" }, { "content_html": "Comments", "url": "https://planetscale.com/blog/its-fine-rewind-revert-a-migration-without-losing-data", "title": "It’s fine, Rewind: Revert a migration without losing data", "date_modified": "2022-03-24T12:16:03.000Z" }, { "content_html": "Comments", "url": "https://tea.xyz/", "title": "Tea – the toolkit that builds the Internet", "date_modified": "2022-03-23T14:42:01.000Z" }, { "content_html": "Comments", "url": "https://github.com/egoist/dum", "title": "Dum: An NPM scripts runner written in Rust", "date_modified": "2022-03-23T13:39:19.000Z" }, { "content_html": "Comments", "url": "https://show-picker.glitch.me/demo.html", "title": "HTMLInputElement ShowPicker()", "date_modified": "2022-03-22T10:46:59.000Z" }, { "content_html": "Comments", "url": "https://den.dev/blog/windows-priority-shuffle/", "title": "Windows needs a change in priorities", "date_modified": "2022-03-20T22:24:22.000Z" }, { "content_html": "Comments", "url": "https://www.seekret.io/blog/ebpf-nuances-on-minikube/", "title": "eBPF Nuances on Minikube", "date_modified": "2022-03-18T12:28:08.000Z" }, { "content_html": "Comments", "url": "https://symflower.com/en/company/blog/2022/complete-guide-on-shrinking-container-images/", "title": "How to make Docker images even smaller", "date_modified": "2022-03-16T19:51:55.000Z" }, { "content_html": "Comments", "url": "https://www.docker.com/blog/speed-boost-achievement-unlocked-on-docker-desktop-4-6-for-mac/", "title": "Speed boost achievement unlocked on Docker Desktop 4.6 for Mac", "date_modified": "2022-03-16T18:00:35.000Z" }, { "content_html": "Comments", "url": "https://tailscale.com/blog/free-plan/", "title": "How our free plan stays free", "date_modified": "2022-03-16T17:14:49.000Z" }, { "content_html": "Comments", "url": "https://ianthehenry.com/posts/how-to-learn-nix/introduction/", "title": "How to Learn Nix", "date_modified": "2022-03-15T02:45:47.000Z" }, { "content_html": "Comments", "url": "https://solito.dev/", "title": "Solito – React Native and Next.js unified", "date_modified": "2022-03-14T20:51:15.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30668137", "title": "Show HN: HN Avatars in 357 bytes", "date_modified": "2022-03-14T03:08:25.000Z" }, { "content_html": "Comments", "url": "https://www.eficode.com/blog/the-future-of-kubernetes-and-why-developers-should-look-beyond-kubernetes-in-2022", "title": "The Future of Kubernetes", "date_modified": "2022-03-13T12:09:05.000Z" }, { "content_html": "Comments", "url": "https://jessicamayzwaan.medium.com/the-legal-implications-of-remote-working-cross-border-64904d274c0d", "title": "The Legal Implications of Remote Working Cross-Border", "date_modified": "2022-03-12T15:50:03.000Z" }, { "content_html": "Comments", "url": "https://contains.dev/blog/mastering-docker-cache", "title": "Mastering the Docker Cache", "date_modified": "2022-03-10T18:04:22.000Z" }, { "content_html": "Comments", "url": "https://frankzliu.com/blog/my-experience-living-and-working-in-china-part-i", "title": "My Experience Working and Living in China", "date_modified": "2022-03-10T08:31:25.000Z" }, { "content_html": "Comments", "url": "https://atlantic.money", "title": "Show HN: World’s first £3 flat fee (0% FX markup) money transfer service", "date_modified": "2022-03-09T07:58:30.000Z" }, { "content_html": "Comments", "url": "https://berkeleygraphics.com/typefaces/berkeley-mono", "title": "Show HN: Berkeley Mono Typeface", "date_modified": "2022-03-04T17:21:25.000Z" }, { "content_html": "Comments", "url": "https://sifted.eu/articles/startup-founders-salary/", "title": "How much do founders pay themselves? A European data set", "date_modified": "2022-03-01T14:01:49.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30497703", "title": "Ask HN: Books you should read when you transform from SWE into SWE-Management", "date_modified": "2022-02-28T08:37:30.000Z" }, { "content_html": "Comments", "url": "https://github.com/orioledb/orioledb", "title": "OrioleDB – solving some PostgreSQL wicked problems", "date_modified": "2022-02-25T01:31:06.000Z" }, { "content_html": "Comments", "url": "https://bytebase.com/blog/how-bytebase-uses-render", "title": "How Bytebase uses Render – to improve dev workflow", "date_modified": "2022-02-24T09:24:58.000Z" }, { "content_html": "Comments", "url": "https://buf.build/blog/buf-cli-v1", "title": "The Buf CLI, an all-in-one tool for Protobuf development, has reached v1.0", "date_modified": "2022-02-23T16:30:46.000Z" }, { "content_html": "

Surviving in Horizon Forbidden West, an open-world game about what happens when Elon Musk funds one too many projects, is only partially contingent on skill. But if you want to thrive in the robo-dino apocalypse, you’ll need to equip yourself with the best gear around.

Read more...

", "url": "https://kotaku.com/horizon-forbidden-west-best-gear-elite-ropecaster-artif-1848578290", "title": "The Best Gear In Horizon Forbidden West (And Where To Find It)", "date_modified": "2022-02-22T21:20:00.000Z" }, { "content_html": "Comments", "url": "https://onlineornot.com/what-learned-running-saas-for-year", "title": "What I learned running a SaaS for a year", "date_modified": "2022-02-22T10:30:19.000Z" }, { "content_html": "Comments", "url": "https://www.kopa.co/blog/posts/how-we-use-notion-as-a-startup", "title": "How we use Notion as a startup", "date_modified": "2022-02-18T23:54:25.000Z" }, { "content_html": "Comments", "url": "https://www.axios.com/qr-code-safety-coinbase-4b7f97d0-940c-45f4-9366-bf5d7f2f3c8f.html", "title": "FBI sounds alarm as QR code usage soars", "date_modified": "2022-02-18T07:32:42.000Z" }, { "content_html": "
\"Production
\"Production

As we develop new products, we often push our operating system - Linux - beyond what is commonly possible. A common theme has been relying on eBPF to build technology that would otherwise have required modifying the kernel. For example, we’ve built DDoS mitigation and a load balancer and use it to monitor our fleet of servers.

This software usually consists of a small-ish eBPF program written in C, executed in the context of the kernel, and a larger user space component that loads the eBPF into the kernel and manages its lifecycle. We’ve found that the ratio of eBPF code to userspace code differs by an order of magnitude or more. We want to shed some light on the issues that a developer has to tackle when dealing with eBPF and present our solutions for building rock-solid production ready applications which contain eBPF.

For this purpose we are open sourcing the production tooling we’ve built for the sk_lookup hook we contributed to the Linux kernel, called tubular. It exists because we’ve outgrown the BSD sockets API. To deliver some products we need features that are just not possible using the standard API.

The source code for tubular is at github.com/cloudflare/tubular, and it allows you to do all the things mentioned above. Maybe the most interesting feature is that you can change the addresses of a service on the fly:

\n

How tubular works

tubular sits at a critical point in the Cloudflare stack, since it has to inspect every connection terminated by a server and decide which application should receive it.

\"Production

Failure to do so will drop or misdirect connections hundreds of times per second. So it has to be incredibly robust during day to day operations. We had the following goals for tubular:

In the past we had built a proof-of-concept control plane for sk_lookup called inet-tool, which proved that we could get away without a persistent service managing the eBPF. Similarly, tubular has tubectl: short-lived invocations make the necessary changes and persisting state is handled by the kernel in the form of eBPF maps. Following this design gave us crash resiliency by default, but left us with the task of mapping the user interface we wanted to the tools available in the eBPF ecosystem.

The tubular user interface

tubular consists of a BPF program that attaches to the sk_lookup hook in the kernel and userspace Go code which manages the BPF program. The tubectl command wraps both in a way that is easy to distribute.

tubectl manages two kinds of objects: bindings and sockets. A binding encodes a rule against which an incoming packet is matched. A socket is a reference to a TCP or UDP socket that can accept new connections or packets.

Bindings and sockets are \"glued\" together via arbitrary strings called labels. Conceptually, a binding assigns a label to some traffic. The label is then used to find the correct socket.

\"Production

Adding bindings

To create a binding that steers port 80 (aka HTTP) traffic destined for 127.0.0.1 to the label “foo” we use tubectl bind:

$ sudo tubectl bind \"foo\" tcp 127.0.0.1 80\n
\n

Due to the power of sk_lookup we can have much more powerful constructs than the BSD API. For example, we can redirect connections to all IPs in 127.0.0.0/24 to a single socket:

$ sudo tubectl bind \"bar\" tcp 127.0.0.0/24 80\n
\n

A side effect of this power is that it's possible to create bindings that \"overlap\":

1: tcp 127.0.0.1/32 80 -> \"foo\"\n2: tcp 127.0.0.0/24 80 -> \"bar\"\n
\n

The first binding says that HTTP traffic to localhost should go to “foo”, while the second asserts that HTTP traffic in the localhost subnet should go to “bar”. This creates a contradiction, which binding should we choose? tubular resolves this by defining precedence rules for bindings:

  1. A prefix with a longer mask is more specific, e.g. 127.0.0.1/32 wins over 127.0.0.0/24.
  2. A port is more specific than the port wildcard, e.g. port 80 wins over \"all ports\" (0).

Applying this to our example, HTTP traffic to all IPs in 127.0.0.0/24 will be directed to foo, except for 127.0.0.1 which goes to bar.

Getting ahold of sockets

sk_lookup needs a reference to a TCP or a UDP socket to redirect traffic to it. However, a socket is usually accessible only by the process which created it with the socket syscall. For example, an HTTP server creates a TCP listening socket bound to port 80. How can we gain access to the listening socket?

A fairly well known solution is to make processes cooperate by passing socket file descriptors via SCM_RIGHTS messages to a tubular daemon. That daemon can then take the necessary steps to hook up the socket with sk_lookup. This approach has several drawbacks:

  1. Requires modifying processes to send SCM_RIGHTS
  2. Requires a tubular daemon, which may crash

There is another way of getting at sockets by using systemd, provided socket activation is used. It works by creating an additional service unit with the correct Sockets setting. In other words: we can leverage systemd oneshot action executed on creation of a systemd socket service, registering the socket into tubular. For example:

[Unit]\nRequisite=foo.socket\n\n[Service]\nType=oneshot\nSockets=foo.socket\nExecStart=tubectl register \"foo\"\n
\n

Since we can rely on systemd to execute tubectl at the correct times we don't need a daemon of any kind. However, the reality is that a lot of popular software doesn't use systemd socket activation. Dealing with systemd sockets is complicated and doesn't invite experimentation. Which brings us to the final trick: pidfd_getfd:

The pidfd_getfd() system call allocates a new file descriptor in the calling process. This new file descriptor is a duplicate of an existing file descriptor, targetfd, in the process referred to by the PID file descriptor pidfd.

We can use it to iterate all file descriptors of a foreign process, and pick the socket we are interested in. To return to our example, we can use the following command to find the TCP socket bound to 127.0.0.1 port 8080 in the httpd process and register it under the \"foo\" label:

$ sudo tubectl register-pid \"foo\" $(pidof httpd) tcp 127.0.0.1 8080\n
\n

It's easy to wire this up using systemd's ExecStartPost if the need arises.

[Service]\nType=forking # or notify\nExecStart=/path/to/some/command\nExecStartPost=tubectl register-pid $MAINPID foo tcp 127.0.0.1 8080\n
\n

Storing state in eBPF maps

As mentioned previously, tubular relies on the kernel to store state, using BPF key / value data structures also known as maps. Using the BPF_OBJ_PIN syscall we can persist them in /sys/fs/bpf:

/sys/fs/bpf/4026532024_dispatcher\n├── bindings\n├── destination_metrics\n├── destinations\n├── sockets\n└── ...\n
\n

The way the state is structured differs from how the command line interface presents it to users. Labels like “foo” are convenient for humans, but they are of variable length. Dealing with variable length data in BPF is cumbersome and slow, so the BPF program never references labels at all. Instead, the user space code allocates numeric IDs, which are then used in the BPF. Each ID represents a (label, domain, protocol) tuple, internally called destination.

For example, adding a binding for \"foo\" tcp 127.0.0.1 ... allocates an ID for (\"foo\", AF_INET, TCP). Including domain and protocol in the destination allows simpler data structures in the BPF. Each allocation also tracks how many bindings reference a destination so that we can recycle unused IDs. This data is persisted into the destinations hash table, which is keyed by (Label, Domain, Protocol) and contains (ID, Count). Metrics for each destination are tracked in destination_metrics in the form of per-CPU counters.

\"Production

bindings is a longest prefix match (LPM) trie which stores a mapping from (protocol, port, prefix) to (ID, prefix length). The ID is used as a key to the sockets map which contains pointers to kernel socket structures. IDs are allocated in a way that makes them suitable as an array index, which allows using the simpler BPF sockmap (an array) instead of a socket hash table. The prefix length is duplicated in the value to work around shortcomings in the BPF API.

\"Production

Encoding the precedence of bindings

As discussed, bindings have a precedence associated with them. To repeat the earlier example:

1: tcp 127.0.0.1/32 80 -> \"foo\"\n2: tcp 127.0.0.0/24 80 -> \"bar\"\n
\n

The first binding should be matched before the second one. We need to encode this in the BPF somehow. One idea is to generate some code that executes the bindings in order of specificity, a technique we’ve used to great effect in l4drop:

1: if (mask(ip, 32) == 127.0.0.1) return \"foo\"\n2: if (mask(ip, 24) == 127.0.0.0) return \"bar\"\n...\n
\n

This has the downside that the program gets longer the more bindings are added, which slows down execution. It's also difficult to introspect and debug such long programs. Instead, we use a specialised BPF longest prefix match (LPM) map to do the hard work. This allows inspecting the contents from user space to figure out which bindings are active, which is very difficult if we had compiled bindings into BPF. The LPM map uses a trie behind the scenes, so lookup has complexity proportional to the length of the key instead of linear complexity for the “naive” solution.

However, using a map requires a trick for encoding the precedence of bindings into a key that we can look up. Here is a simplified version of this encoding, which ignores IPv6 and uses labels instead of IDs. To insert the binding tcp 127.0.0.0/24 80 into a trie we first convert the IP address into a number.

127.0.0.0    = 0x7f 00 00 00\n
\n

Since we're only interested in the first 24 bits of the address we, can write the whole prefix as

127.0.0.0/24 = 0x7f 00 00 ??\n
\n

where “?” means that the value is not specified. We choose the number 0x01 to represent TCP and prepend it and the port number (80 decimal is 0x50 hex) to create the full key:

tcp 127.0.0.0/24 80 = 0x01 50 7f 00 00 ??\n
\n

Converting tcp 127.0.0.1/32 80 happens in exactly the same way. Once the converted values are inserted into the trie, the LPM trie conceptually contains the following keys and values.

LPM trie:\n        0x01 50 7f 00 00 ?? = \"bar\"\n        0x01 50 7f 00 00 01 = \"foo\"\n
\n

To find the binding for a TCP packet destined for 127.0.0.1:80, we again encode a key and perform a lookup.

input:  0x01 50 7f 00 00 01   TCP packet to 127.0.0.1:80\n---------------------------\nLPM trie:\n        0x01 50 7f 00 00 ?? = \"bar\"\n           y  y  y  y  y\n        0x01 50 7f 00 00 01 = \"foo\"\n           y  y  y  y  y  y\n---------------------------\nresult: \"foo\"\n\ny = byte matches\n
\n

The trie returns “foo” since its key shares the longest prefix with the input. Note that we stop comparing keys once we reach unspecified “?” bytes, but conceptually “bar” is still a valid result. The distinction becomes clear when looking up the binding for a TCP packet to 127.0.0.255:80.

input:  0x01 50 7f 00 00 ff   TCP packet to 127.0.0.255:80\n---------------------------\nLPM trie:\n        0x01 50 7f 00 00 ?? = \"bar\"\n           y  y  y  y  y\n        0x01 50 7f 00 00 01 = \"foo\"\n           y  y  y  y  y  n\n---------------------------\nresult: \"bar\"\n\nn = byte doesn't match\n
\n

In this case \"foo\" is discarded since the last byte doesn't match the input. However, \"bar\" is returned since its last byte is unspecified and therefore considered to be a valid match.

Observability with minimal privileges

Linux has the powerful ss tool (part of iproute2) available to inspect socket state:

$ ss -tl src 127.0.0.1\nState      Recv-Q      Send-Q           Local Address:Port           Peer Address:Port\nLISTEN     0           128                  127.0.0.1:ipp                 0.0.0.0:*\n
\n

With tubular in the picture this output is not accurate anymore. tubectl bindings makes up for this shortcoming:

$ sudo tubectl bindings tcp 127.0.0.1\nBindings:\n protocol       prefix port label\n      tcp 127.0.0.1/32   80   foo\n
\n

Running this command requires super-user privileges, despite in theory being safe for any user to run. While this is acceptable for casual inspection by a human operator, it's a dealbreaker for observability via pull-based monitoring systems like Prometheus. The usual approach is to expose metrics via an HTTP server, which would have to run with elevated privileges and be accessible to the Prometheus server somehow. Instead, BPF gives us the tools to enable read-only access to tubular state with minimal privileges.

The key is to carefully set file ownership and mode for state in /sys/fs/bpf. Creating and opening files in /sys/fs/bpf uses BPF_OBJ_PIN and BPF_OBJ_GET. Calling BPF_OBJ_GET with BPF_F_RDONLY is roughly equivalent to open(O_RDONLY) and allows accessing state in a read-only fashion, provided the file permissions are correct. tubular gives the owner full access but restricts read-only access to the group:

$ sudo ls -l /sys/fs/bpf/4026532024_dispatcher | head -n 3\ntotal 0\n-rw-r----- 1 root root 0 Feb  2 13:19 bindings\n-rw-r----- 1 root root 0 Feb  2 13:19 destination_metrics\n
\n

It's easy to choose which user and group should own state when loading tubular:

$ sudo -u root -g tubular tubectl load\ncreated dispatcher in /sys/fs/bpf/4026532024_dispatcher\nloaded dispatcher into /proc/self/ns/net\n$ sudo ls -l /sys/fs/bpf/4026532024_dispatcher | head -n 3\ntotal 0\n-rw-r----- 1 root tubular 0 Feb  2 13:42 bindings\n-rw-r----- 1 root tubular 0 Feb  2 13:42 destination_metrics\n
\n

There is one more obstacle, systemd mounts /sys/fs/bpf in a way that makes it inaccessible to anyone but root. Adding the executable bit to the directory fixes this.

$ sudo chmod -v o+x /sys/fs/bpf\nmode of '/sys/fs/bpf' changed from 0700 (rwx------) to 0701 (rwx-----x)\n
\n

Finally, we can export metrics without privileges:

$ sudo -u nobody -g tubular tubectl metrics 127.0.0.1 8080\nListening on 127.0.0.1:8080\n^C\n
\n

There is a caveat, unfortunately: truly unprivileged access requires unprivileged BPF to be enabled. Many distros have taken to disabling it via the unprivileged_bpf_disabled sysctl, in which case scraping metrics does require CAP_BPF.

Safe releases

tubular is distributed as a single binary, but really consists of two pieces of code with widely differing lifetimes. The BPF program is loaded into the kernel once and then may be active for weeks or months, until it is explicitly replaced. In fact, a reference to the program (and link, see below) is persisted into /sys/fs/bpf:

/sys/fs/bpf/4026532024_dispatcher\n├── link\n├── program\n└── ...\n
\n

The user space code is executed for seconds at a time and is replaced whenever the binary on disk changes. This means that user space has to be able to deal with an \"old\" BPF program in the kernel somehow. The simplest way to achieve this is to compare what is loaded into the kernel with the BPF shipped as part of tubectl. If the two don't match we return an error:

$ sudo tubectl bind foo tcp 127.0.0.1 80\nError: bind: can't open dispatcher: loaded program #158 has differing tag: \"938c70b5a8956ff2\" doesn't match \"e007bfbbf37171f0\"\n
\n

tag is the truncated hash of the instructions making up a BPF program, which the kernel makes available for every loaded program:

$ sudo bpftool prog list id 158\n158: sk_lookup  name dispatcher  tag 938c70b5a8956ff2\n...\n
\n

By comparing the tag tubular asserts that it is dealing with a supported version of the BPF program. Of course, just returning an error isn't enough. There needs to be a way to update the kernel program so that it's once again safe to make changes. This is where the persisted link in /sys/fs/bpf comes into play. bpf_links are used to attach programs to various BPF hooks. \"Enabling\" a BPF program is a two-step process: first, load the BPF program, next attach it to a hook using a bpf_link. Afterwards the program will execute the next time the hook is executed. By updating the link we can change the program on the fly, in an atomic manner.

$ sudo tubectl upgrade\nUpgraded dispatcher to 2022.1.0-dev, program ID #159\n$ sudo bpftool prog list id 159\n159: sk_lookup  name dispatcher  tag e007bfbbf37171f0\n…\n$ sudo tubectl bind foo tcp 127.0.0.1 80\nbound foo#tcp:[127.0.0.1/32]:80\n
\n

Behind the scenes the upgrade procedure is slightly more complicated, since we have to update the pinned program reference in addition to the link. We pin the new program into /sys/fs/bpf:

/sys/fs/bpf/4026532024_dispatcher\n├── link\n├── program\n├── program-upgrade\n└── ...\n
\n

Once the link is updated we atomically rename program-upgrade to replace program. In the future we may be able to use RENAME_EXCHANGE to make upgrades even safer.

Preventing state corruption

So far we’ve completely neglected the fact that multiple invocations of tubectl could modify the state in /sys/fs/bpf at the same time. It’s very hard to reason about what would happen in this case, so in general it’s best to prevent this from ever occurring. A common solution to this is advisory file locks. Unfortunately it seems like BPF maps don't support locking.

$ sudo flock /sys/fs/bpf/4026532024_dispatcher/bindings echo works!\nflock: cannot open lock file /sys/fs/bpf/4026532024_dispatcher/bindings: Input/output error\n
\n

This led to a bit of head scratching on our part. Luckily it is possible to flock the directory instead of individual maps:

$ sudo flock --exclusive /sys/fs/bpf/foo echo works!\nworks!\n
\n

Each tubectl invocation likewise invokes flock(), thereby guaranteeing that only ever a single process is making changes.

Conclusion

tubular is in production at Cloudflare today and has simplified the deployment of Spectrum and our authoritative DNS. It allowed us to leave behind limitations of the BSD socket API. However, its most powerful feature is that the addresses a service is available on can be changed on the fly. In fact, we have built tooling that automates this process across our global network. Need to listen on another million IPs on thousands of machines? No problem, it’s just an HTTP POST away.

Interested in working on tubular and our L4 load balancer unimog? We are hiring in our European offices.

", "url": "https://blog.cloudflare.com/tubular-fixing-the-socket-api-with-ebpf/", "title": "Production ready eBPF, or how we fixed the BSD socket API", "date_modified": "2022-02-17T17:02:54.000Z" }, { "content_html": "Comments", "url": "https://future.a16z.com/software-development-building-for-99-developers/", "title": "Building for the 99% Developers", "date_modified": "2022-02-17T00:19:16.000Z" }, { "content_html": "Comments", "url": "https://docs.temporal.io/blog/series-b-announcement-open-letter/", "title": "An open letter to the Temporal user community", "date_modified": "2022-02-16T20:32:50.000Z" }, { "content_html": "Comments", "url": "https://www.gosecure.net/blog/2022/02/14/current-mfa-fatigue-attack-campaign-targeting-microsoft-office-365-users/", "title": "Current MFA fatigue attack campaign targeting Microsoft Office 365 users", "date_modified": "2022-02-16T18:17:07.000Z" }, { "content_html": "Comments", "url": "https://developer.1password.com/docs/ssh/", "title": "1Password for SSH and Git", "date_modified": "2022-02-16T13:01:17.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30329762", "title": "Ask HN: What made your business take off that you wish you'd done much earlier?", "date_modified": "2022-02-14T08:47:58.000Z" }, { "content_html": "Comments", "url": "https://ardentperf.com/2022/02/10/a-hairy-postgresql-incident/", "title": "A Hairy PostgreSQL Incident", "date_modified": "2022-02-11T03:22:45.000Z" }, { "content_html": "Comments", "url": "https://www.edgedb.com/blog/edgedb-1-0", "title": "Show HN: EdgeDB 1.0", "date_modified": "2022-02-10T18:13:03.000Z" }, { "content_html": "Comments", "url": "https://blog.crunchydata.com/blog/postgres-constraints-for-newbies", "title": "Postgres Constraints for Newbies", "date_modified": "2022-02-09T22:25:28.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/our-user-mode-wireguard-year/", "title": "Our User-Mode WireGuard Year", "date_modified": "2022-02-09T18:06:38.000Z" }, { "content_html": "Comments", "url": "https://brandur.org/fragments/single-dependency-stacks", "title": "Single dependency stacks", "date_modified": "2022-02-09T16:53:29.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30272101", "title": "Ask HN: What's your solution for SSL on internal servers?", "date_modified": "2022-02-09T13:00:34.000Z" }, { "content_html": "Comments", "url": "https://github.com/ariga/atlas", "title": "Atlas – A Database Toolkit", "date_modified": "2022-02-08T09:00:29.000Z" }, { "content_html": "Comments", "url": "https://webonastick.com/fonts/routed-gothic/", "title": "Routed Gothic Font", "date_modified": "2022-02-03T09:16:27.000Z" }, { "content_html": "Comments", "url": "https://chown.me/blog/getting-my-own-asn", "title": "Why and How I Got My Own ASN", "date_modified": "2022-02-02T15:15:23.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30178571", "title": "Show HN: Just Launched an App for Dads", "date_modified": "2022-02-02T15:09:27.000Z" }, { "content_html": "

Comments

", "url": "https://chown.me/blog/getting-my-own-asn", "title": "Why and how I got my own ASN", "date_modified": "2022-02-02T12:37:13.000Z" }, { "content_html": "Comments", "url": "https://databricks.com/blog/2022/01/26/creating-a-faster-tar-extractor.html", "title": "Speeding up LXC container pull by up to 3x", "date_modified": "2022-02-01T17:34:46.000Z" }, { "content_html": "Comments", "url": "https://northflank.com/", "title": "Northflank – Simplifying application deployment to cloud platforms", "date_modified": "2022-02-01T13:00:54.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/theryanking/status/1487500943511932941", "title": "“Y Combinator is not worth it”", "date_modified": "2022-01-29T19:30:49.000Z" }, { "content_html": "Comments", "url": "https://www.quora.com/Why-do-tech-startups-still-hire-so-many-people-Given-the-well-documented-costs-both-direct-and-indirect-of-increasing-the-size-of-organizations-why-do-so-many-tech-companies-still-grow-so-quickly", "title": "Why do startups hire so many people?", "date_modified": "2022-01-28T21:27:45.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=30102240", "title": "Ask HN: Hacker claimed ownership and then deleted my Facebook Page of 50k users", "date_modified": "2022-01-27T16:09:19.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/run-ordinary-rails-apps-globally/", "title": "Run Ordinary Rails Apps Globally", "date_modified": "2022-01-26T10:19:59.000Z" }, { "content_html": "Comments", "url": "https://thewhyaxis.substack.com/p/how-paul-giamatti-broke-the-california", "title": "Paul Giamatti broke the California wine industry", "date_modified": "2022-01-25T02:53:04.000Z" }, { "content_html": "Comments", "url": "https://github.com/madler/zlib", "title": "Zlib – a spiffy yet delicately unobtrusive compression library", "date_modified": "2022-01-24T09:00:27.000Z" }, { "content_html": "Comments", "url": "https://charm.sh/", "title": "Charm – tools to make the command line glamorous", "date_modified": "2022-01-23T17:46:12.000Z" }, { "content_html": "Comments", "url": "https://github.com/khuedoan/homelab", "title": "My self-hosting infrastructure, fully automated", "date_modified": "2022-01-21T22:52:34.000Z" }, { "content_html": "Comments", "url": "https://atlasgo.io/", "title": "Atlas – Terraform but for Database Migrations", "date_modified": "2022-01-21T07:23:27.000Z" }, { "content_html": "Comments", "url": "https://spacelift.io/blog/tricking-postgres-into-using-query-plan", "title": "Tricking PostgreSQL into using an insane, but faster, query plan", "date_modified": "2022-01-18T16:42:24.000Z" }, { "content_html": "Comments", "url": "https://pawelurbanek.com/postgresql-query-bottleneck", "title": "PostgreSQL query performance bottlenecks", "date_modified": "2022-01-18T11:58:13.000Z" }, { "content_html": "Comments", "url": "https://www.shayon.dev/post/2022/17/why-i-enjoy-postgresql-infrastructure-engineers-perspective/", "title": "Why I Enjoy PostgreSQL – Infrastructure Engineer's Perspective", "date_modified": "2022-01-17T21:10:17.000Z" }, { "content_html": "Comments", "url": "https://research.nccgroup.com/2022/01/13/10-real-world-stories-of-how-weve-compromised-ci-cd-pipelines/", "title": "Real-world stories of how we’ve compromised CI/CD pipelines", "date_modified": "2022-01-17T11:08:01.000Z" }, { "content_html": "Comments", "url": "https://github.com/disposable-email-domains/disposable-email-domains/pull/298", "title": "Mozilla's Firefox Relay to be added to disposable-email-domains blacklist", "date_modified": "2022-01-16T19:28:42.000Z" }, { "content_html": "Comments", "url": "https://www.outseta.com/posts/five-years-a-founder", "title": "Five Years a [Bootstrapped] Founder", "date_modified": "2022-01-15T04:07:29.000Z" }, { "content_html": "
\"A
\"A

Recently, we made an optimization to the Cloudflare Workers runtime which reduces the amount of time Workers need to spend in memory. We're passing the savings on to you for all your Unbound Workers.

Background

Workers are often used to implement HTTP proxies, where JavaScript is used to rewrite an HTTP request before sending it on to an origin server, and then to rewrite the response before sending it back to the client. You can implement any kind of rewrite in a Worker, including both rewriting headers and bodies.

Many Workers, though, do not actually modify the response body, but instead simply allow the bytes to pass through from the origin to the client. In this case, the Worker's application code has finished executing as soon as the response headers are sent, before the body bytes have passed through. Historically, the Worker was nevertheless considered to be \"in use\" until the response body had fully finished streaming.

For billing purposes, under the Workers Unbound pricing model, we charge duration-memory (gigabyte-seconds) for the time in which the Worker is in use.

The change

On December 15-16, we made a change to the way we handle requests that are streaming through the response without modifying the content. This change means that we can mark application code as “idle” as soon as the response headers are returned.

Since no further application code will execute on behalf of the request, the system does not need to keep the request state in memory – it only needs to track the low-level native sockets and pump the bytes through. So now, during this time, the Worker will be considered idle, and could even be evicted before the stream completes (though this would be unlikely unless the stream lasts for a very long time).

Visualized it looks something like this:

\"A

As a result of this change, we've seen that the time a Worker is considered \"in use\" by any particular request has dropped by an average of 70%. Of course, this number varies a lot depending on the details of each Worker. Some may see no benefit, others may see an even larger benefit.

This change is totally invisible to the application. To any external observer, everything behaves as it did before. But, since the system now considers a Worker to be idle during response streaming, the response streaming time will no longer be billed. So, if you saw a drop in your bill, this is why!

But it doesn’t stop there!

The change also applies to a few other frequently used scenarios, namely Websocket proxying, reading from the cache and streaming from KV.

WebSockets: once a Worker has arranged to proxy through a WebSocket, as long as it isn't handling individual messages in your Worker code, the Worker does not remain in use during the proxying. The change applies to regular stateless Workers, but not to Durable Objects, which are not usually used for proxying.

export default {\n  async fetch(request: Request) {\n    //Do anything before\n    const upgradeHeader = request.headers.get('Upgrade')\n    if (upgradeHeader || upgradeHeader === 'websocket') {\n      return await fetch(request)\n    }\n    //Or with other requests\n  }\n}\n
\n

Reading from Cache: If you return the response from a cache.match call, the Worker is considered idle as soon as the response headers are returned.

export default {\n  async fetch(request: Request) {\n    let response = await caches.default.match('https://example.com')\n    if (response) {\n      return response\n    }\n    // get/create response and put into cache\n  }\n}\n
\n

Streaming from KV: And lastly, when you stream from KV. This one is a bit trickier to get right, because often people retrieve the value from KV as a string, or JSON object and then create a response with that value. But if you fetch the value as a stream, as done in the example below, you can create a Response with the ReadableStream.

interface Env {\n  MY_KV_NAME: KVNamespace\n}\n\nexport default {\n  async fetch(request: Request, env: Env) {\n    const readableStream = await env.MY_KV_NAME.get('hello_world.pdf', { type: 'stream' })\n    if (readableStream) {\n      return new Response(readableStream, { headers: { 'content-type': 'application/pdf' } })\n    }\n  },\n}\n
\n

Interested in Workers Unbound?

If you are already using Unbound, your bill will have automatically dropped already.

Now is a great time to check out Unbound if you haven’t already, especially since recently, we’ve also removed the egress fees. Unbound allows you to build more complex workloads on our platform and only pay for what you use.

We are always looking for opportunities to make Workers better. Often that improvement takes the form of powerful new features such as the soon-to-be released Service Bindings and, of course, performance enhancements. This time, we are delighted to make Cloudflare Workers even cheaper than they already were.

", "url": "https://blog.cloudflare.com/workers-optimization-reduces-your-bill/", "title": "A Workers optimization\nthat reduces your bill", "date_modified": "2022-01-14T13:58:51.000Z" }, { "content_html": "Comments", "url": "https://arxiv.org/abs/2201.00223", "title": "They Still Haven't Told You", "date_modified": "2022-01-12T20:47:38.000Z" }, { "content_html": "Comments", "url": "https://olickel.com/button-syndrome", "title": "Losing our product to button syndrome", "date_modified": "2022-01-12T19:56:15.000Z" }, { "content_html": "Comments", "url": "https://mullvad.net/en/blog/2022/1/12/diskless-infrastructure-beta-system-transparency-stboot/", "title": "Mullvad: Diskless infrastructure using stboot in beta", "date_modified": "2022-01-12T08:10:01.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29892472", "title": "Ask HN: Why isn't there a backlash around charging for security features?", "date_modified": "2022-01-11T15:20:53.000Z" }, { "content_html": "Comments", "url": "https://www.bolt.com/blog/switching-four-day-workweek/#", "title": "Switching to a four-day workweek (2021)", "date_modified": "2022-01-10T17:43:10.000Z" }, { "content_html": "Comments", "url": "https://blog.ycombinator.com/ycs-standard-deal/", "title": "YC’s $500k Standard Deal", "date_modified": "2022-01-10T17:36:16.000Z" }, { "content_html": "Comments", "url": "https://blog.sequin.io/events-not-webhooks", "title": "Give me /events, not webhooks", "date_modified": "2022-01-08T01:12:43.000Z" }, { "content_html": "Comments", "url": "https://contains.dev/blog/optimizing-docker-image-size", "title": "Optimizing Docker image size and why it matters", "date_modified": "2022-01-06T19:13:01.000Z" }, { "content_html": "Comments", "url": "https://jvns.ca/blog/2017/06/28/notes-on-bpf---ebpf/", "title": "Notes on BPF and eBPF", "date_modified": "2022-01-02T19:48:23.000Z" }, { "content_html": "Comments", "url": "https://joeldare.com/why-im-using-http-basic-auth-in-2022.html", "title": "Why I'm Using HTTP Basic Auth in 2022", "date_modified": "2022-01-01T19:25:59.000Z" }, { "content_html": "Comments", "url": "https://fusionauth.io/learn/expert-advice/oauth/modern-guide-to-oauth/", "title": "The Modern Guide to OAuth", "date_modified": "2021-12-31T21:54:00.000Z" }, { "content_html": "Comments", "url": "https://medium.com/bigpanda-engineering/what-is-engineering-enablement-b5293a5838ce", "title": "What Is Engineering Enablement", "date_modified": "2021-12-31T18:48:52.000Z" }, { "content_html": "

Official project URL www.benthos.dev/

\n\n

Comments

", "url": "https://twitter.com/Jeffail/status/1476598958445285384", "title": "Benthos, the Swiss Army knife of stream processing, reached 100 contributors", "date_modified": "2021-12-31T13:10:30.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29733026", "title": "Ask HN: What’s the Most Outrageous Belief You’re Confident Is True?", "date_modified": "2021-12-30T02:57:49.000Z" }, { "content_html": "Comments", "url": "https://coda.io/@mykola-bilokonsky/public-neurodiversity-support-center/how-to-talk-about-autism-respectfully-84", "title": "How to Talk About Autism Respectfully", "date_modified": "2021-12-29T22:40:19.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29723455", "title": "Ask HN: What are some good rust code to read to learn the language?", "date_modified": "2021-12-29T10:08:43.000Z" }, { "content_html": "Comments", "url": "https://danluu.com/cgroup-throttling/", "title": "The Container Throttling Problem", "date_modified": "2021-12-26T08:42:48.000Z" }, { "content_html": "Comments", "url": "https://ptribble.blogspot.com/2021/12/the-cost-of-cloud.html", "title": "The Cost of Cloud", "date_modified": "2021-12-25T23:16:42.000Z" }, { "content_html": "Comments", "url": "https://github.com/galvez/typejuice", "title": "Typejuice: Docs generator for .d.ts files inspired by godoc", "date_modified": "2021-12-25T05:21:37.000Z" }, { "content_html": "Comments", "url": "https://adhdatwork.add.org/adhd-accommodation-guide/", "title": "ADHD Accommodations Guide", "date_modified": "2021-12-24T23:57:35.000Z" }, { "content_html": "Comments", "url": "https://tinyssh.org", "title": "Tinyssh", "date_modified": "2021-12-23T12:27:09.000Z" }, { "content_html": "Comments", "url": "https://tuple.app/", "title": "Tuple: Pair Programming Tool for macOS", "date_modified": "2021-12-21T20:46:19.000Z" }, { "content_html": "Comments", "url": "https://airbnb.io/lottie/", "title": "Lottie – Use after effects animations in web and native apps", "date_modified": "2021-12-21T05:08:48.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29633404", "title": "Ask HN: Recommended Domain Registrars?", "date_modified": "2021-12-21T03:15:30.000Z" }, { "content_html": "Comments", "url": "https://github.com/haimgel/display-switch", "title": "Display-switch: Turn a $30 USB switch into a full-featured multi-monitor KVM", "date_modified": "2021-12-18T22:00:29.000Z" }, { "content_html": "Comments", "url": "https://webapp.io/blog/postgres-is-the-answer/", "title": "Postgres is a great pub/sub and job server (2019)", "date_modified": "2021-12-17T22:27:35.000Z" }, { "content_html": "Comments", "url": "https://help.sunsama.com/docs/pricing-manifesto", "title": "SaaS Pricing Manifesto", "date_modified": "2021-12-16T04:14:12.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29563226", "title": "Ask HN: How do you manage direct updates to databases in a production system", "date_modified": "2021-12-15T08:07:20.000Z" }, { "content_html": "Comments", "url": "https://dynomight.net/plans/", "title": "Plans you're not supposed to talk about", "date_modified": "2021-12-14T17:34:37.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29523638", "title": "Ask HN: Long-form info-dense videos on YouTube you would recommend to anyone?", "date_modified": "2021-12-11T19:44:57.000Z" }, { "content_html": "Comments", "url": "https://buf.build/blog/an-update-on-our-fundraising", "title": "Buf raises $93M to deprecate REST/JSON", "date_modified": "2021-12-08T20:20:42.000Z" }, { "content_html": "Comments", "url": "https://leejo.github.io/2021/12/08/curiosities-in-vinyl/", "title": "Curiosities in Vinyl", "date_modified": "2021-12-08T07:41:54.000Z" }, { "content_html": "Comments", "url": "https://martinfowler.com/articles/cant-buy-integration.html", "title": "You Can't Buy Integration", "date_modified": "2021-12-07T21:20:19.000Z" }, { "content_html": "Comments", "url": "https://www.flyertalk.com/forum/33719083-post2040.html", "title": "Google 20% time volunteers have been rewriting the ITA Matrix flight search app", "date_modified": "2021-12-03T01:47:43.000Z" }, { "content_html": "Comments", "url": "http://jpetazzo.github.io/2021/11/30/docker-build-container-images-antipatterns/", "title": "Anti-patterns when building container images", "date_modified": "2021-11-30T13:30:12.000Z" }, { "content_html": "Comments", "url": "https://chriskiehl.com/article/event-sourcing-is-hard", "title": "Event Sourcing Is Hard", "date_modified": "2021-11-30T09:41:26.000Z" }, { "content_html": "Comments", "url": "https://postgrest.org/en/v9.0/releases/v9.0.0.html", "title": "PostgREST v9.0.0", "date_modified": "2021-11-30T06:38:17.000Z" }, { "content_html": "Comments", "url": "https://blog.asciinema.org/post/smaller-faster/", "title": "4x Smaller, 50x Faster", "date_modified": "2021-11-30T01:40:22.000Z" }, { "content_html": "

Today we are announcing that Karpenter is ready for production. Karpenter is an open-source, flexible, high-performance Kubernetes cluster autoscaler built with AWS. It helps improve your application availability and cluster efficiency by rapidly launching right-sized compute resources in response to changing application load. Karpenter also provides just-in-time compute resources to meet your application’s needs and will soon automatically optimize a cluster’s compute resource footprint to reduce costs and improve performance.

\n

Before Karpenter, Kubernetes users needed to dynamically adjust the compute capacity of their clusters to support applications using Amazon EC2 Auto Scaling groups and the Kubernetes Cluster Autoscaler. Nearly half of Kubernetes customers on AWS report that configuring cluster auto scaling using the Kubernetes Cluster Autoscaler is challenging and restrictive.

\n

When Karpenter is installed in your cluster, Karpenter observes the aggregate resource requests of unscheduled pods and makes decisions to launch new nodes and terminate them to reduce scheduling latencies and infrastructure costs. Karpenter does this by observing events within the Kubernetes cluster and then sending commands to the underlying cloud provider’s compute service, such as Amazon EC2.

\n

\n

Karpenter is an open-source project licensed under the Apache License 2.0. It is designed to work with any Kubernetes cluster running in any environment, including all major cloud providers and on-premises environments. We welcome contributions to build additional cloud providers or to improve core project functionality. If you find a bug, have a suggestion, or have something to contribute, please engage with us on GitHub.

\n

Getting Started with Karpenter on AWS
To get started with Karpenter in any Kubernetes cluster, ensure there is some compute capacity available, and install it using the Helm charts provided in the public repository. Karpenter also requires permissions to provision compute resources on the provider of your choice.

\n

Once installed in your cluster, the default Karpenter provisioner will observe incoming Kubernetes pods, which cannot be scheduled due to insufficient compute resources in the cluster and automatically launch new resources to meet their scheduling and resource requirements.

\n

I want to show a quick start using Karpenter in an Amazon EKS cluster based on Getting Started with Karpenter on AWS. It requires the installation of AWS Command Line Interface (AWS CLI), kubectl, eksctl, and Helm (the package manager for Kubernetes). After setting up these tools, create a cluster with eksctl. This example configuration file specifies a basic cluster with one initial node.

\n
cat <<EOF > cluster.yaml\n---\napiVersion: eksctl.io/v1alpha5\nkind: ClusterConfig\nmetadata:\n  name: eks-karpenter-demo\n  region: us-east-1\n  version: \"1.20\"\nmanagedNodeGroups:\n  - instanceType: m5.large\n    amiFamily: AmazonLinux2\n    name: eks-kapenter-demo-ng\n    desiredCapacity: 1\n    minSize: 1\n    maxSize: 5\nEOF\n$ eksctl create cluster -f cluster.yaml
\n

Karpenter itself can run anywhere, including on self-managed node groups, managed node groups, or AWS Fargate. Karpenter will provision EC2 instances in your account.

\n

Next, you need to create necessary AWS Identity and Access Management (IAM) resources using the AWS CloudFormation template and IAM Roles for Service Accounts (IRSA) for the Karpenter controller to get permissions like launching instances following the documentation. You also need to install the Helm chart to deploy Karpenter to your cluster.

\n
$ helm repo add karpenter https://charts.karpenter.sh\n$ helm repo update\n$ helm upgrade --install --skip-crds karpenter karpenter/karpenter --namespace karpenter \\\n  --create-namespace --set serviceAccount.create=false --version 0.5.0 \\\n  --set controller.clusterName=eks-karpenter-demo\n  --set controller.clusterEndpoint=$(aws eks describe-cluster --name eks-karpenter-demo --query \"cluster.endpoint\" --output json) \\\n  --wait # for the defaulting webhook to install before creating a Provisioner
\n

Karpenter provisioners are a Kubernetes resource that enables you to configure the behavior of Karpenter in your cluster. When you create a default provisioner, without further customization besides what is needed for Karpenter to provision compute resources in your cluster, Karpenter automatically discovers node properties such as instance types, zones, architectures, operating systems, and purchase types of instances. You don’t need to define these spec:requirements if there is no explicit business requirement.

\n
cat <<EOF | kubectl apply -f -\napiVersion: karpenter.sh/v1alpha5\nkind: Provisioner\nmetadata:\nname: default\nspec:\n#Requirements that constrain the parameters of provisioned nodes. \n#Operators { In, NotIn } are supported to enable including or excluding values\n  requirements:\n    - key: node.k8s.aws/instance-type #If not included, all instance types are considered\n      operator: In\n      values: [\"m5.large\", \"m5.2xlarge\"]\n    - key: \"topology.kubernetes.io/zone\" #If not included, all zones are considered\n      operator: In\n      values: [\"us-east-1a\", \"us-east-1b\"]\n    - key: \"kubernetes.io/arch\" #If not included, all architectures are considered\n      values: [\"arm64\", \"amd64\"]\n    - key: \" karpenter.sh/capacity-type\" #If not included, the webhook for the AWS cloud provider will default to on-demand\n      operator: In\n      values: [\"spot\", \"on-demand\"]\n  provider:\n    instanceProfile: KarpenterNodeInstanceProfile-eks-karpenter-demo\n  ttlSecondsAfterEmpty: 30  \nEOF
\n

The ttlSecondsAfterEmpty value configures Karpenter to terminate empty nodes. If this value is disabled, nodes will never scale down due to low utilization. To learn more, see Provisioner custom resource definitions (CRDs) on the Karpenter site.

\n

Karpenter is now active and ready to begin provisioning nodes in your cluster. Create some pods using a deployment, and watch Karpenter provision nodes in response.

\n
$ kubectl create deployment --name inflate \\\n          --image=public.ecr.aws/eks-distro/kubernetes/pause:3.2
\n

Let’s scale the deployment and check out the logs of the Karpenter controller.

\n
$ kubectl scale deployment inflate --replicas 10\n$ kubectl logs -f -n karpenter $(kubectl get pods -n karpenter -l karpenter=controller -o name)\n2021-11-23T04:46:11.280Z        INFO    controller.allocation.provisioner/default       Starting provisioning loop      {\"commit\": \"abc12345\"}\n2021-11-23T04:46:11.280Z        INFO    controller.allocation.provisioner/default       Waiting to batch additional pods        {\"commit\": \"abc123456\"}\n2021-11-23T04:46:12.452Z        INFO    controller.allocation.provisioner/default       Found 9 provisionable pods      {\"commit\": \"abc12345\"}\n2021-11-23T04:46:13.689Z        INFO    controller.allocation.provisioner/default       Computed packing for 10 pod(s) with instance type option(s) [m5.large]  {\"commit\": \" abc123456\"}\n2021-11-23T04:46:16.228Z        INFO    controller.allocation.provisioner/default       Launched instance: i-01234abcdef, type: m5.large, zone: us-east-1a, hostname: ip-192-168-0-0.ec2.internal    {\"commit\": \"abc12345\"}\n2021-11-23T04:46:16.265Z        INFO    controller.allocation.provisioner/default       Bound 9 pod(s) to node ip-192-168-0-0.ec2.internal  {\"commit\": \"abc12345\"}\n2021-11-23T04:46:16.265Z        INFO    controller.allocation.provisioner/default       Watching for pod events {\"commit\": \"abc12345\"}
\n

The provisioner’s controller listens for Pods changes, which launched a new instance and bound the provisionable Pods into the new nodes.

\n

Now, delete the deployment. After 30 seconds (ttlSecondsAfterEmpty = 30), Karpenter should terminate the empty nodes.

\n
$ kubectl delete deployment inflate\n$ kubectl logs -f -n karpenter $(kubectl get pods -n karpenter -l karpenter=controller -o name)\n2021-11-23T04:46:18.953Z        INFO    controller.allocation.provisioner/default       Watching for pod events {\"commit\": \"abc12345\"}\n2021-11-23T04:49:05.805Z        INFO    controller.Node Added TTL to empty node ip-192-168-0-0.ec2.internal {\"commit\": \"abc12345\"}\n2021-11-23T04:49:35.823Z        INFO    controller.Node Triggering termination after 30s for empty node ip-192-168-0-0.ec2.internal {\"commit\": \"abc12345\"}\n2021-11-23T04:49:35.849Z        INFO    controller.Termination  Cordoned node ip-192-168-116-109.ec2.internal   {\"commit\": \"abc12345\"}\n2021-11-23T04:49:36.521Z        INFO    controller.Termination  Deleted node ip-192-168-0-0.ec2.internal    {\"commit\": \"abc12345\"}
\n

If you delete a node with kubectl, Karpenter will gracefully cordon, drain, and shut down the corresponding instance. Under the hood, Karpenter adds a finalizer to the node object, which blocks deletion until all pods are drained, and the instance is terminated.

\n

Things to Know
Here are a couple of things to keep in mind about Kapenter features:

\n

Accelerated Computing: Karpenter works with all kinds of Kubernetes applications, but it performs particularly well for use cases that require rapid provisioning and deprovisioning large numbers of diverse compute resources quickly. For example, this includes batch jobs to train machine learning models, run simulations, or perform complex financial calculations. You can leverage custom resources of nvidia.com/gpu, amd.com/gpu, and aws.amazon.com/neuron for use cases that require accelerated EC2 instances.

\n

Provisioners Compatibility: Kapenter provisioners are designed to work alongside static capacity management solutions like Amazon EKS managed node groups and EC2 Auto Scaling groups. You may choose to manage the entirety of your capacity using provisioners, a mixed model with both dynamic and statically managed capacity, or a fully static approach. We recommend not using Kubernetes Cluster Autoscaler at the same time as Karpenter because both systems scale up nodes in response to unschedulable pods. If configured together, both systems will race to launch or terminate instances for these pods.

\n

Join our Karpenter Community
Karpenter’s community is open to everyone. Give it a try, and join our working group meeting for future releases that interest you. As I said, we welcome any contributions such as bug reports, new features, corrections, or additional documentation.

\n

To learn more about Karpenter, see the documentation and demo video from AWS Container Day.

\n

– Channy

\n \n
", "url": "https://aws.amazon.com/blogs/aws/introducing-karpenter-an-open-source-high-performance-kubernetes-cluster-autoscaler/", "title": "Introducing Karpenter – An Open-Source High-Performance Kubernetes Cluster Autoscaler", "date_modified": "2021-11-30T00:45:44.000Z" }, { "content_html": "Comments", "url": "https://blog.jetbrains.com/blog/2021/11/29/welcome-to-fleet/", "title": "Fleet, a Lightweight IDE from JetBrains", "date_modified": "2021-11-29T08:20:59.000Z" }, { "content_html": "Comments", "url": "https://microsoft.github.io/code-with-engineering-playbook/ENG-FUNDAMENTALS-CHECKLIST/", "title": "Code with Engineering Playbook", "date_modified": "2021-11-27T15:46:14.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29340015", "title": "Ask HN: Tips on Sysadmin Job", "date_modified": "2021-11-25T10:24:06.000Z" }, { "content_html": "Comments", "url": "https://ory.dev/login-spa-react-nextjs-authentication-example-api-open-source/", "title": "Show HN: Open-Source Auth for NextJS / React", "date_modified": "2021-11-24T13:08:09.000Z" }, { "content_html": "Comments", "url": "https://www.mindtheproduct.com/overengineering-can-kill-your-product/", "title": "Overengineering can kill your product", "date_modified": "2021-11-24T10:53:18.000Z" }, { "content_html": "
\n \"the\n
Photo: James Bareham/Polygon
\n
\n\n

Defunctland’s new video goes deeper on crowd management than you ever thought possible

\n

\n Continue reading…\n

", "url": "https://www.polygon.com/22798637/disney-fastpass-2021-disneyland-disney-world-defunctland", "title": "This is how Disney parks’ Fastpass spun completely out of control", "date_modified": "2021-11-23T22:00:00.000Z" }, { "content_html": "Comments", "url": "https://wasp-lang.dev/blog/2021/11/22/fundraising-learnings", "title": "Our fundraising learning post-YC", "date_modified": "2021-11-23T20:51:19.000Z" }, { "content_html": "Comments", "url": "https://sifted.eu/articles/brexit-london-new-york-leaving/", "title": "I’m leaving London for NYC and taking my tech startup", "date_modified": "2021-11-23T10:43:37.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29297229", "title": "Ask HN: Great tools for solo SaaS founders?", "date_modified": "2021-11-21T16:08:59.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29264754", "title": "Ask HN: What makes a good engineering blog?", "date_modified": "2021-11-18T12:42:42.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29264374", "title": "Ask HN: What are you using for public documentation these days?", "date_modified": "2021-11-18T11:43:39.000Z" }, { "content_html": "Comments", "url": "https://github.com/Jarred-Sumner/hop", "title": "Hop: Faster than unzip and tar at reading individual files", "date_modified": "2021-11-10T18:50:46.000Z" }, { "content_html": "Comments", "url": "https://www.apple.com/business/essentials/", "title": "Apple Business Essentials", "date_modified": "2021-11-10T15:03:06.000Z" }, { "content_html": "Comments", "url": "https://hakibenita.com/postgresql-unknown-features", "title": "Lesser Known PostgreSQL Features", "date_modified": "2021-11-09T15:50:35.000Z" }, { "content_html": "Comments", "url": "https://rosefinchapp.com", "title": "Show HN: Insomnia-like client for SQLite", "date_modified": "2021-11-08T17:52:00.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=29099746", "title": "Ask HN: Who's not sucky to work for?", "date_modified": "2021-11-03T20:19:08.000Z" }, { "content_html": "Comments", "url": "https://eisel.me/devtool-allocators", "title": "Faster Mac dev tools with custom allocators", "date_modified": "2021-11-01T16:18:20.000Z" }, { "content_html": "Comments", "url": "https://medium.com/swlh/the-codeless-cto-471ee6069288", "title": "The Codeless CTO", "date_modified": "2021-10-27T20:04:01.000Z" }, { "content_html": "Comments", "url": "https://heap.io/blog/how-postgres-audit-tables-saved-us-from-taking-down-production", "title": "Postgres audit tables saved us from taking down production", "date_modified": "2021-10-26T17:32:19.000Z" }, { "content_html": "Comments", "url": "https://www.osohq.com/post/cross-platform-rust-libraries", "title": "We Built a Cross-Platform Library with Rust", "date_modified": "2021-10-25T20:02:39.000Z" }, { "content_html": "Comments", "url": "https://www.koyeb.com/blog/building-a-multi-region-service-mesh-with-kuma-envoy-anycast-bgp-and-mtls", "title": "Building a Multi-Region Service Mesh", "date_modified": "2021-10-25T12:29:01.000Z" }, { "content_html": "Comments", "url": "https://blog.battlefy.com/how-not-to-blow-up-the-production-database-424c162dccc6?gi=8ec446ad5e47", "title": "How not to blow up the production database", "date_modified": "2021-10-15T18:23:10.000Z" }, { "content_html": "

Comments

", "url": "https://github.com/CISOfy/lynis", "title": "Lynis - Security auditing tool for Linux, macOS, and UNIX-based systems", "date_modified": "2021-10-13T21:56:09.000Z" }, { "content_html": "Comments", "url": "https://sysdig.com/blog/container-security-best-practices/", "title": "Container security best practices: Ultimate guide", "date_modified": "2021-10-13T16:47:47.000Z" }, { "content_html": "Comments", "url": "https://github.com/maxgoedjen/secretive", "title": "Secretive: An app for storing and managing SSH keys in the Secure Enclave", "date_modified": "2021-10-13T15:22:50.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=28838132", "title": "Ask HN: Solo-preneurs, how do you DevOps to save time?", "date_modified": "2021-10-12T10:35:11.000Z" }, { "content_html": "Comments", "url": "https://www.simplethread.com/20-things-ive-learned-in-my-20-years-as-a-software-engineer/", "title": "Things I’ve learned in my 20 years as a software engineer", "date_modified": "2021-10-08T10:04:32.000Z" }, { "content_html": "Comments", "url": "https://www.notion.so/blog/sharding-postgres-at-notion", "title": "Lessons learned from sharding Postgres at Notion", "date_modified": "2021-10-06T18:50:42.000Z" }, { "content_html": "Comments", "url": "https://authzed.com/blog/spicedb-is-open-source/", "title": "SpiceDB Is Open Source", "date_modified": "2021-09-30T18:58:34.000Z" }, { "content_html": "Comments", "url": "https://boringseo.org/", "title": "Boring SEO guide for the non-lazy", "date_modified": "2021-09-30T14:15:31.000Z" }, { "content_html": "Comments", "url": "https://www.fastmail.com/1password/", "title": "Masked Email from Fastmail and 1Password", "date_modified": "2021-09-28T12:33:17.000Z" }, { "content_html": "
\n \"a\n
Graphic: James Bareham/Polygon | Source images: Netflix
\n
\n\n\n \t\t

‘[Adult animation] is ripe for new kinds of formats and new types of stories’

\n

\n Continue reading…\n

", "url": "https://www.polygon.com/animation-cartoons/22696218/inside-job-netflix-creator-shion-takeuchi-gravity-falls-interview", "title": "Gravity Falls writer Shion Takeuchi on her latest creation: Netflix’s Inside Job", "date_modified": "2021-09-27T16:40:00.000Z" }, { "content_html": "Comments", "url": "https://building.nubank.com.br/why-we-killed-our-end-to-end-test-suite/", "title": "We killed our end-to-end test suite", "date_modified": "2021-09-24T15:36:53.000Z" }, { "content_html": "Comments", "url": "https://shufflesharding.com/posts/aws-sigv4-and-sigv4a", "title": "AWS SIGv4 and SIGv4A – How AWS signs and verifies API requests", "date_modified": "2021-09-23T19:56:51.000Z" }, { "content_html": "Comments", "url": "https://github.com/gmpetrov/utlimate-saas-js", "title": "The ultimate SaaS template", "date_modified": "2021-09-23T13:14:03.000Z" }, { "content_html": "Comments", "url": "https://motion.dev/", "title": "Show HN: Motion One, the Web Animations API for Everyone", "date_modified": "2021-09-22T13:27:10.000Z" }, { "content_html": "Comments", "url": "https://rabexc.org/posts/pitfalls-of-ssh-agents", "title": "The pitfalls of using SSH-agent, or how to use an agent safely", "date_modified": "2021-09-18T14:40:49.000Z" }, { "content_html": "Comments", "url": "https://shkspr.mobi/blog/2021/09/how-to-add-issn-metadata-to-a-web-page/", "title": "How to add ISSN metadata to a web page", "date_modified": "2021-09-17T11:13:33.000Z" }, { "content_html": "Comments", "url": "https://blog.px.dev/ebpf-openssl-tracing/", "title": "Tracing SSL/TLS connections using eBPF", "date_modified": "2021-09-16T17:44:03.000Z" }, { "content_html": "Comments", "url": "https://microfounder.com/blog/cofounder-in-marketing", "title": "Developer, you may need a co-founder in marketing", "date_modified": "2021-09-16T16:26:19.000Z" }, { "content_html": "Comments", "url": "https://linuxplumbersconf.org/event/11/contributions/954/", "title": "Use of eBPF in CPU Scheduler", "date_modified": "2021-09-16T13:34:27.000Z" }, { "content_html": "Comments", "url": "https://awsteele.com/blog/2021/09/15/aws-federation-comes-to-github-actions.html", "title": "AWS federation comes to GitHub Actions", "date_modified": "2021-09-15T03:44:54.000Z" }, { "content_html": "Comments", "url": "https://github.com/containerd/nerdctl", "title": "Docker compatible open source: Containerd nerdctl", "date_modified": "2021-09-05T21:41:40.000Z" }, { "content_html": "Comments", "url": "https://matklad.github.io/2021/09/04/fast-rust-builds.html", "title": "Fast Rust Builds", "date_modified": "2021-09-05T15:36:01.000Z" }, { "content_html": "Comments", "url": "https://psychedelicsociety.org.uk/news/bw06yvjnvblq47vwoocv5owchiyf8b", "title": "Do psychedelics make you liberal? Not always", "date_modified": "2021-09-04T13:46:42.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=28299053", "title": "Ask HN: Companies of one, what is your tech stack?", "date_modified": "2021-08-25T07:54:35.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/api-tokens-a-tedious-survey/", "title": "API Tokens: A Tedious Survey", "date_modified": "2021-08-24T21:41:02.000Z" }, { "content_html": "Comments", "url": "https://ramp.com/blog/ramp-finance-automation-platform", "title": "Ramp raises $300M", "date_modified": "2021-08-24T17:11:51.000Z" }, { "content_html": "Comments", "url": "https://sysprog21.github.io/lkmpg/", "title": "Linux Kernel Module Programming Guide", "date_modified": "2021-08-23T22:49:01.000Z" }, { "content_html": "Comments", "url": "https://pop.com/home", "title": "Show HN: Pop.com – pair programming with low-latency, Screenhero-style sharing", "date_modified": "2021-08-18T16:25:01.000Z" }, { "content_html": "Comments", "url": "https://www.theguardian.com/us-news/2021/aug/16/guantanamo-detainee-mansoor-adayfi", "title": "The US should be held accountable: Gitmo survivor on the war on terror’s failure", "date_modified": "2021-08-16T07:25:08.000Z" }, { "content_html": "Comments", "url": "https://sigstore.dev", "title": "Sigstore – A new standard for signing, verifying and protecting software", "date_modified": "2021-08-09T19:14:22.000Z" }, { "content_html": "Comments", "url": "https://www.brendangregg.com/blog/2019-08-19/bpftrace.html", "title": "A thorough introduction to bpftrace", "date_modified": "2021-08-09T05:33:28.000Z" }, { "content_html": "Comments", "url": "https://hookdeck.com?ref=hn", "title": "Show HN: Hookdeck – an infrastructure to consume webhooks", "date_modified": "2021-08-04T17:14:50.000Z" }, { "content_html": "Comments", "url": "https://www.banterly.net/2021/07/31/new-in-git-switch-and-restore/", "title": "New in Git: switch and restore", "date_modified": "2021-08-01T09:11:57.000Z" }, { "content_html": "

Comments

", "url": "https://security.googleblog.com/2021/06/verifiable-supply-chain-metadata-for.html", "title": "Verifiable Supply Chain Metadata for Tekton", "date_modified": "2021-07-13T15:29:02.000Z" }, { "content_html": "Comments", "url": "https://earthly.dev/", "title": "Show HN: Earthly – Better Builds", "date_modified": "2021-07-09T16:21:10.000Z" }, { "content_html": "Comments", "url": "https://runwayml.com/", "title": "Runway – Create Impossible Video", "date_modified": "2021-07-07T22:49:53.000Z" }, { "content_html": "Comments", "url": "https://brendangregg.com/blog/2021-07-03/how-to-add-bpf-observability.html", "title": "How to add eBPF observability to your product", "date_modified": "2021-07-03T16:50:07.000Z" }, { "content_html": "Comments", "url": "https://jez.io/pandoc-markdown-css-theme/", "title": "Show HN: Pandoc Markdown CSS Theme", "date_modified": "2021-06-27T07:20:58.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=27579693", "title": "Ask HN: I was hit with a patent troll lawsuit, how do I deal with it?", "date_modified": "2021-06-21T14:48:27.000Z" }, { "content_html": "Comments", "url": "https://www.lrb.co.uk/the-paper/v43/n12/sharon-marcus/more-husband-than-female", "title": "More Husband Than Female", "date_modified": "2021-06-20T05:16:10.000Z" }, { "content_html": "Comments", "url": "https://brendangregg.com/blog/2021-06-04/an-unbelievable-demo.html", "title": "An Unbelievable Demo", "date_modified": "2021-06-04T04:21:31.000Z" }, { "content_html": "Comments", "url": "https://lwn.net/SubscriberLink/858023/1caabaef50d4946b/", "title": "Auditing Io_uring", "date_modified": "2021-06-03T14:56:11.000Z" }, { "content_html": "Comments", "url": "https://old.reddit.com/r/ExperiencedDevs/comments/nmodyl/drunk_post_things_ive_learned_as_a_sr_engineer/", "title": "Drunk Post: Things I've Learned as a Sr Engineer", "date_modified": "2021-05-30T13:55:24.000Z" }, { "content_html": "Comments", "url": "https://boringavatars.com/", "title": "Boring Avatars – react library to generate custom avatars", "date_modified": "2021-05-28T05:15:48.000Z" }, { "content_html": "Comments", "url": "https://www.marcolancini.it/2021/blog-cloud-security-roadmap/", "title": "On Establishing a Cloud Security Program", "date_modified": "2021-05-19T19:49:01.000Z" }, { "content_html": "Comments", "url": "https://www.planetscale.com/blog/announcing-planetscale-the-database-for-developers", "title": "PlanetScale – Database for Developers", "date_modified": "2021-05-18T17:16:54.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=27192884", "title": "Ask HN: Desperately need “sales for nerds” advice", "date_modified": "2021-05-18T10:04:42.000Z" }, { "content_html": "Comments", "url": "https://www.apple.com/newsroom/2021/05/apple-music-announces-spatial-audio-and-lossless-audio/", "title": "Apple Music Announces Spatial Audio and Lossless Audio", "date_modified": "2021-05-17T13:06:31.000Z" }, { "content_html": "

Comments

", "url": "https://arnon.dk/5-things-i-learned-developing-billing-system/", "title": "5 things I learned while developing a billing system", "date_modified": "2021-05-15T17:22:55.000Z" }, { "content_html": "Comments", "url": "https://www.theguardian.com/lifeandstyle/2019/nov/06/i-was-an-astrologer-how-it-works-psychics", "title": "I was an astrologer – here's how it really works", "date_modified": "2021-05-15T16:31:39.000Z" }, { "content_html": "Comments", "url": "https://awslabs.github.io/smithy/", "title": "Smithy: A language for defining services and SDKs", "date_modified": "2021-05-07T21:31:58.000Z" }, { "content_html": "Comments", "url": "https://meta.stackexchange.com/questions/364048/were-switching-to-system-fonts-on-may-10-2021", "title": "Stack Overflow switches to system fonts on May 10, 2021", "date_modified": "2021-05-06T14:43:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/255kb/stack-on-a-budget", "title": "Stack on a Budget – A collection of services with free tiers", "date_modified": "2021-05-03T06:01:26.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2021-04-29-scaling-monorepo-maintenance/", "title": "Scaling Monorepo Maintenance at GitHub", "date_modified": "2021-04-30T09:52:08.000Z" }, { "content_html": "Comments", "url": "https://medium.com/airbnb-engineering/our-journey-towards-cloud-efficiency-9c02ba04ade8", "title": "Our Journey Towards Cloud Efficiency", "date_modified": "2021-04-27T20:21:51.000Z" }, { "content_html": "Comments", "url": "https://console.dev/qa/keydb-john-sully/", "title": "KeyDB CEO Interview: Getting into YC with a Fork of Redis", "date_modified": "2021-04-27T15:01:47.000Z" }, { "content_html": "Comments", "url": "https://github.com/RobinCsl/awesome-js-tooling-not-in-js", "title": "JavaScript Tooling Not in JavaScript", "date_modified": "2021-04-20T09:24:00.000Z" }, { "content_html": "Comments", "url": "https://developers.google.com/style/word-list", "title": "Word usage guidance and alternative terms", "date_modified": "2021-04-19T08:15:16.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=26856387", "title": "Ask HN: How do you run better meetings?", "date_modified": "2021-04-18T21:08:33.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=26856053", "title": "Ask HN: Considering colocated hosting over cloud, what should I know?", "date_modified": "2021-04-18T20:32:54.000Z" }, { "content_html": "

Comments

", "url": "https://rhardih.io/2021/04/listing-the-contents-of-a-remote-zip-archive-without-downloading-the-entire-file/", "title": "Listing the contents of a remote ZIP archive, without downloading the entire file", "date_modified": "2021-04-18T15:47:05.000Z" }, { "content_html": "Comments", "url": "https://itamargilad.com/product-led/", "title": "Product-Led Is Just as Bad as Sales-Driven", "date_modified": "2021-04-18T10:58:51.000Z" }, { "content_html": "Comments", "url": "https://jakedeichert.com/blog/reducing-rust-incremental-compilation-times-on-macos-by-70-percent/", "title": "Reducing Rust Incremental Compilation Times on macOS by 70%", "date_modified": "2021-04-18T07:59:15.000Z" }, { "content_html": "Comments", "url": "https://madned.substack.com/p/thanks-for-the-bonus-i-quit", "title": "Thanks for the Bonus, I Quit", "date_modified": "2021-04-17T21:55:59.000Z" }, { "content_html": "Comments", "url": "https://fly.io/blog/docker-without-docker/", "title": "Docker without Docker", "date_modified": "2021-04-09T03:06:39.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2021-04-05-how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/", "title": "We scaled the GitHub API with a sharded, replicated rate limiter in Redis", "date_modified": "2021-04-08T13:26:42.000Z" }, { "content_html": "Comments", "url": "https://anthonynsimon.com/blog/one-man-saas-architecture/", "title": "The Architecture of a One-Man SaaS", "date_modified": "2021-04-08T12:14:01.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/", "title": "Behind GitHub’s new authentication token formats", "date_modified": "2021-04-05T16:32:58.000Z" }, { "content_html": "Comments", "url": "https://mmartinfahy.medium.com/my-experience-releasing-3-failed-saas-products-44e61cbde424", "title": "My experience releasing 3 failed SaaS products", "date_modified": "2021-04-04T02:10:09.000Z" }, { "content_html": "Comments", "url": "https://airbyte.io/articles/our-story/the-deck-we-used-to-raise-our-seed-with-accel-in-13-days/", "title": "The deck we used to raise our seed", "date_modified": "2021-03-31T17:47:34.000Z" }, { "content_html": "Comments", "url": "http://www.pathsensitive.com/2021/03/developer-tools-can-be-magic-instead.html", "title": "Developer tools can be magic but instead collect dust", "date_modified": "2021-03-28T18:34:03.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2021-03-25-how-github-actions-renders-large-scale-logs/", "title": "How GitHub Actions renders large-scale logs", "date_modified": "2021-03-25T16:30:03.000Z" }, { "content_html": "Comments", "url": "http://dan.bodar.com/2012/02/28/crazy-fast-build-times-or-when-10-seconds-starts-to-make-you-nervous/", "title": "Crazy fast build times, or when 10 seconds starts to make you nervous (2012)", "date_modified": "2021-03-24T05:44:30.000Z" }, { "content_html": "Comments", "url": "https://www.lennysnewsletter.com/p/saas-pricing-strategy", "title": "How to price your SaaS product", "date_modified": "2021-03-23T11:16:51.000Z" }, { "content_html": "Comments", "url": "https://magicbell.io/blog/the-ycombinator-experience-remote-edition", "title": "The Y Combinator Experience – Remote Edition", "date_modified": "2021-03-16T08:49:02.000Z" }, { "content_html": "Comments", "url": "https://superuser.com/questions/1633073/why-are-tar-xz-files-15x-smaller-when-using-pythons-tar-library-compared-to-mac", "title": "Why are tar.xz files 15x smaller when using Python's tar compared to macOS tar?", "date_modified": "2021-03-14T21:06:37.000Z" }, { "content_html": "Comments", "url": "https://github.com/gajus/slonik", "title": "Slonik: A PostgreSQL client with strict types, detailed logging, and assertions", "date_modified": "2021-03-10T20:12:23.000Z" }, { "content_html": "Comments", "url": "https://bdickason.com/posts/speed-is-the-killer-feature/", "title": "Speed Is the Killer Feature", "date_modified": "2021-03-02T06:40:48.000Z" }, { "content_html": "Comments", "url": "https://github.com/kuchin/awesome-cto", "title": "Awesome CTO", "date_modified": "2021-02-27T13:37:53.000Z" }, { "content_html": "Comments", "url": "https://www.lesswrong.com/posts/jHnFBHrwiNb5xvLBM/second-citizenships-residencies-and-or-temporary-relocation", "title": "Second Citizenships, Residencies, and/or Temporary Relocation", "date_modified": "2021-02-21T02:07:19.000Z" }, { "content_html": "Comments", "url": "https://lethain.com/managing-staff-plus-engineers/", "title": "Managing Staff-Plus Engineers", "date_modified": "2021-02-16T14:23:15.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=26144706", "title": "Ask HN: Is “contact us for pricing” a dark pattern?", "date_modified": "2021-02-15T17:16:13.000Z" }, { "content_html": "
\n \"Austin\n
Image: New Line Cinema
\n
\n\n

A few shagedelic recommendations from across Netflix, HBO Max, and Amazon

\n

\n Continue reading…\n

", "url": "https://www.polygon.com/streaming/2021/2/13/22278710/best-new-movies-february-2021-netflix-amazon-hbo-disney-plus", "title": "11 best movies new to streaming to watch in February", "date_modified": "2021-02-13T15:32:00.000Z" }, { "content_html": "Comments", "url": "https://nofreeplan.com", "title": "Don't Offer a Free Plan", "date_modified": "2021-02-08T00:26:40.000Z" }, { "content_html": "Comments", "url": "https://chriskiehl.com/article/thoughts-after-6-years", "title": "Software engineering topics I changed my mind on", "date_modified": "2021-01-24T00:02:48.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=25686678", "title": "Ask HN: Which companies work like Gumroad?", "date_modified": "2021-01-08T16:56:03.000Z" }, { "content_html": "Comments", "url": "https://compiledcssinjs.com/", "title": "Show HN: Compiled, buildtime atomic CSS in JavaScript and all your favorite APIs", "date_modified": "2021-01-02T06:49:09.000Z" }, { "content_html": "Comments", "url": "https://francescodilorenzo.com/saas-we-pay-for", "title": "SaaS We Happily Pay For", "date_modified": "2020-12-27T16:59:53.000Z" }, { "content_html": "Comments", "url": "https://github.com/dpapathanasiou/simple-graph", "title": "Show HN: Simple-graph – a graph database in SQLite", "date_modified": "2020-12-26T16:31:47.000Z" }, { "content_html": "

Comments

", "url": "https://dev.to/baweaver/tales-of-the-autistic-developer-the-expert-pk2", "title": "Tales of the Autistic Developer - The Expert", "date_modified": "2020-12-26T03:47:07.000Z" }, { "content_html": "

Comments

", "url": "http://calculist.org/blog/2020/12/21/tac/", "title": "Toolchains as Code", "date_modified": "2020-12-21T17:12:00.000Z" }, { "content_html": "

Comments

", "url": "https://blog.volta.sh/2020/12/21/announcing-volta-100/", "title": "Announcing Volta 1.0.0", "date_modified": "2020-12-21T17:11:36.000Z" }, { "content_html": "Comments", "url": "https://leanpub.com/implementing-ddd-cqrs-and-event-sourcing", "title": "Show HN: I wrote a book on implementing DDD, CQRS and Event Sourcing", "date_modified": "2020-12-19T22:24:46.000Z" }, { "content_html": "Comments", "url": "https://www.bloomberg.com/news/articles/2020-12-18/czech-startup-founders-turn-billionaires-without-vc-help", "title": "Jetbrains founders turn billionaires without VC help", "date_modified": "2020-12-18T12:06:06.000Z" }, { "content_html": "

Comments

", "url": "https://buttondown.email/nelhage/archive/papers-i-love-gg/", "title": "Papers I love: gg", "date_modified": "2020-12-06T17:22:06.000Z" }, { "content_html": "Comments", "url": "https://clig.dev/", "title": "CLI Guidelines – A guide to help you write better command-line programs", "date_modified": "2020-12-04T16:33:36.000Z" }, { "content_html": "Comments", "url": "https://github.com/turbo/pg-shortkey", "title": "pg-shortkey: YouTube-Like Short IDs as Postgres Primary Keys", "date_modified": "2020-12-03T12:26:11.000Z" }, { "content_html": "Comments", "url": "https://archive.org/details/attentionkmartshoppers", "title": "Attention K-Mart Shoppers: Piped Music Collection on the Internet Archive", "date_modified": "2020-12-01T23:31:04.000Z" }, { "content_html": "Comments", "url": "https://about.sourcegraph.com/blog/ex-googler-guide-dev-tools/", "title": "An ex-Googler’s guide to dev tools", "date_modified": "2020-11-26T05:59:28.000Z" }, { "content_html": "Comments", "url": "https://linkerd.io/2020/11/23/topology-aware-service-routing-on-kubernetes-with-linkerd/", "title": "Topology-Aware Service Routing on Kubernetes with Linkerd", "date_modified": "2020-11-25T19:41:14.000Z" }, { "content_html": "Comments", "url": "https://pingcap.com/blog/run-database-on-aws-with-better-performance-and-lower-cost/", "title": "How to Run a Database on AWS with Better Performance and Lower Cost", "date_modified": "2020-11-25T02:18:40.000Z" }, { "content_html": "

Earlier this year, we announced Cloud Load Balancer support for Cloud Run. You might wonder, aren't Cloud Run services already load-balanced? Yes, each *.run.app endpoint load balances traffic between an autoscaling set of containers. However, with the Cloud Balancing integration for serverless platforms, you can now fine tune lower levels of your networking stack. In this article, we will explain the use cases for this type of set up and build an HTTPS load balancer from ground up for Cloud Run using Terraform.

Why use a Load Balancer for Cloud Run?

Every Cloud Run service comes with a load-balanced *.run.app endpoint that’s secured with HTTPS. Furthermore, Cloud Run also lets you map your custom domains to your services. However, if you want to customize other details about how your load balancing works, you need to provision a Cloud HTTP load balancer yourself.

Here are a few reasons to run your Cloud Run service behind a Cloud Load Balancer:

The list goes on, Cloud HTTP Load Balancing has quite a lot of features.

Why use Terraform for this?

The short answer is that a Cloud HTTP Load Balancer consists of many networking resources that you need to create and connect to each other. There’s no single \"load balancer\" object in GCP APIs.

To understand the upcoming task, let's take a look at the resources involved:

As you might imagine, it is very tedious to provision and connect these resources just to achieve a simple task like enabling CDN.

You could write a bash script with the gcloud command-line tool to create these resources; however, it will be cumbersome to check corner cases like if a resource already exists, or modified manually later. You would also need to write a cleanup script to delete what you provisioned.

This is where Terraform shines. It lets you declaratively configure cloud resources and create/destroy your stack in different GCP projects efficiently with just a few commands.

Building a load balancer: The hard way

The goal of this article is to intentionally show you the hard way for each resource involved in creating a load balancer using Terraform configuration language.

We'll start with a few Terraform variables:

First, let's define our Terraform providers:

Then, let's deploy a new Cloud Run service named \"hello\" with the sample image, and allow unauthenticated access to it:

If you manage your Cloud Run deployments outside Terraform, that’s perfectly fine: You can still import the equivalent data source to reference that service in your configuration file.

Next, we’ll reserve a global IPv4 address for our global load balancer:

Next, let's create a managed SSL certificate that's issued and renewed by Google for you:

If you want to bring your own SSL certificates, you can create your own google_compute_ssl_certificate resource instead.

Then, make a network endpoint group (NEG) out of your serverless service:

Now, let's create a backend service that'll keep track of these network endpoints:

If you want to configure load balancing features such as CDN, Cloud Armor or custom headers, the google_compute_backend_service resource is the right place.

Then, create an empty URL map that doesn't have any routing rules and sends the traffic to this backend service we created earlier:

Next, configure an HTTPS proxy to terminate the traffic with the Google-managed certificate and route it to the URL map:

Finally, configure a global forwarding rule to route the HTTPS traffic on the IP address to the target HTTPS proxy:

After writing this module, create an output variable that lists your IP address:

When you apply these resources and set your domain’s DNS records to point to this IP address, a huge machinery starts rolling its wheels.

Soon, Google Cloud will verify your domain name ownership and start to issue a managed TLS certificate for your domain. After the certificate is issued, the load balancer configuration will propagate to all of Google’s edge locations around the globe. This might take a while, but once it starts working.

Astute readers will notice that so far this setup cannot handle the unencrypted HTTP traffic. Therefore, any requests that come over port 80 are dropped, which is not great for usability. To mitigate this, you need to create a new set of URL map, target HTTP proxy, and a forwarding rule with these:

As we are nearing 150 lines of Terraform configuration, you probably have realized by now, this is indeed the hard way to get a load balancer for your serverless applications.

If you like to try out this example, feel free to obtain a copy of this Terraform configuration file from this gist and adopt it for your needs.

Building a load balancer: The easy way

To address the complexity in this experience, we have been designing a new Terraform module specifically to skip the hard parts of deploying serverless applications behind a Cloud HTTPS Load Balancer.

Stay tuned for the next article where we take a closer look at this new Terraform module and show you how easier this can get.

Related Article

Global HTTP(S) Load Balancing and CDN now support serverless compute

Now, our App Engine, Cloud Run and Cloud Functions serverless compute offerings can take advantage of global load balancing and Cloud CDN.

Read Article
", "url": "https://cloud.google.com/blog/topics/developers-practitioners/serverless-load-balancing-terraform-hard-way/", "title": "Serverless load balancing with Terraform: The hard way", "date_modified": "2020-11-24T18:00:00.000Z" }, { "content_html": "Comments", "url": "https://aws.amazon.com/blogs/opensource/why-aws-loves-rust-and-how-wed-like-to-help/", "title": "Why AWS loves Rust, and how we’d like to help", "date_modified": "2020-11-24T17:00:57.000Z" }, { "content_html": "Comments", "url": "https://github.com/microsoft/TypeScript/wiki/Performance", "title": "Tips for Performant TypeScript", "date_modified": "2020-11-24T15:06:08.000Z" }, { "content_html": "Comments", "url": "https://adamfard.com/blog/enterprise-ux", "title": "Enterprise UX Design: Make me think", "date_modified": "2020-11-24T10:07:17.000Z" }, { "content_html": "Comments", "url": "https://www.netice9.com/blog/guide-to-oomkill-alerting-in-kubernetes-clusters", "title": "Guide to OOMKill Alerting in Kubernetes Clusters", "date_modified": "2020-11-23T22:31:31.000Z" }, { "content_html": "Comments", "url": "https://panelbear.com/blog/tech-stack/", "title": "The Tech Stack of a One-Man SaaS", "date_modified": "2020-11-23T13:06:53.000Z" }, { "content_html": "Comments", "url": "https://www.ox.ac.uk/news/2020-11-23-oxford-university-breakthrough-global-covid-19-vaccine", "title": "Oxford University breakthrough on global COVID-19 vaccine", "date_modified": "2020-11-23T07:25:28.000Z" }, { "content_html": "Comments", "url": "https://foundersatwork.posthaven.com/startups-the-very-beginning", "title": "Startups: The Beginning", "date_modified": "2020-11-21T17:27:01.000Z" }, { "content_html": "Comments", "url": "https://vale.dev/", "title": "The Vale Programming Language", "date_modified": "2020-11-20T13:25:07.000Z" }, { "content_html": "Comments", "url": "https://github.com/artipie/artipie", "title": "New binary artifact management tool", "date_modified": "2020-11-20T10:33:35.000Z" }, { "content_html": "Comments", "url": "https://github.com/PostgREST/postgrest", "title": "PostgREST: REST API for any Postgres database", "date_modified": "2020-11-20T10:25:36.000Z" }, { "content_html": "Comments", "url": "https://devfonts.gafi.dev/", "title": "Dev Fonts", "date_modified": "2020-11-20T10:14:44.000Z" }, { "content_html": "Comments", "url": "https://devblogs.microsoft.com/typescript/announcing-typescript-4-1/", "title": "TypeScript 4.1", "date_modified": "2020-11-19T23:28:50.000Z" }, { "content_html": "Comments", "url": "https://blog.tailwindcss.com/tailwindcss-v2", "title": "TailwindCSS v2.0", "date_modified": "2020-11-18T18:42:06.000Z" }, { "content_html": "Comments", "url": "https://macosicons.com/", "title": "macOS Icons", "date_modified": "2020-11-17T15:17:36.000Z" }, { "content_html": "Comments", "url": "https://changelog.com/posts/git-is-simply-too-hard", "title": "Git is too hard", "date_modified": "2020-11-17T08:43:20.000Z" }, { "content_html": "Comments", "url": "https://www.userinterviewexchange.com/blog/user-interview-template", "title": "Show HN: I created a user interview template", "date_modified": "2020-11-17T00:39:59.000Z" }, { "content_html": "Comments", "url": "https://www.themvpsprint.com/p/how-and-when-to-acquire-saas-users", "title": "How and when to acquire SaaS users", "date_modified": "2020-11-16T17:15:43.000Z" }, { "content_html": "Comments", "url": "https://tomcam.github.io/postgres/", "title": "PostgreSQL psql command line tutorial and cheat sheet", "date_modified": "2020-11-16T16:18:56.000Z" }, { "content_html": "

Comments

", "url": "https://jvns.ca/blog/2020/11/15/simple-explanations-without-sounding-condescending/", "title": "How do you write simple explanations without sounding condescending?", "date_modified": "2020-11-16T02:52:50.000Z" }, { "content_html": "Comments", "url": "https://linc.sh/blog/durable-objects-in-production", "title": "Durable Objects in Production", "date_modified": "2020-11-13T17:09:29.000Z" }, { "content_html": "Comments", "url": "https://landshark.io/2020/11/13/saas-needs-a-single-point-of-purchase.html", "title": "SaaS Needs a Single Point of Purchase", "date_modified": "2020-11-13T13:03:06.000Z" }, { "content_html": "Comments", "url": "https://github.com/jemmaissroff/find_github_email", "title": "Show HN: Find GitHub Users' Email Address", "date_modified": "2020-11-12T22:38:11.000Z" }, { "content_html": "Comments", "url": "https://vectorized.io/redpanda", "title": "Redpanda – A Kafka-compatible streaming platform for mission-critical workloads", "date_modified": "2020-11-12T22:04:50.000Z" }, { "content_html": "Comments", "url": "http://koyeb.com/blog/announcing-the-koyeb-serverless-engine-docker-containers-and-continuous-deployment-of-functions", "title": "Koyeb Serverless Engine: Docker Containers, Continuous Deployment of Functions", "date_modified": "2020-11-12T14:47:19.000Z" }, { "content_html": "Comments", "url": "https://blog.m3o.com/2020/11/12/netlify-for-the-frontend-micro-for-the-backend.html", "title": "Netlify for the frontend, Micro for the backend", "date_modified": "2020-11-12T11:44:19.000Z" }, { "content_html": "Comments", "url": "https://www.quantamagazine.org/physicists-pin-down-nuclear-reaction-from-moments-after-the-big-bang-20201111/", "title": "Physicists Pin Down Nuclear Reaction from Moments After the Big Bang", "date_modified": "2020-11-12T04:55:47.000Z" }, { "content_html": "Comments", "url": "https://pgstats.dev/", "title": "Postgres Observability", "date_modified": "2020-11-11T13:12:51.000Z" }, { "content_html": "

A few years ago I was preparing my talk for a conference and couldn’t find a good illustration of Postgres internals. This is how Postgres Observability was created – a diagram that gives a simplified view of all system views and functions for Postgres monitoring and tracking statistics.

\n

I’ve been updating the static version after each release and it got busier and busier overtime, so I decided that it is time to create a more in depth version of it. 

\n

So here is the updated for PostgreSQL 13 interactive version of PostgreSQL Observability Pgstats.dev.

\n

It now has additional information about each internal and each stat. Just follow the links and you can discover more about each one of Postgres components and ways they interact.

\n

Share it! 

\n

The post Interactive new look for PostgreSQL Observability appeared first on Data Egret.

", "url": "https://postgr.es/p/4Xn", "title": "Alexey Lesovsky: Interactive new look for PostgreSQL Observability", "date_modified": "2020-11-10T10:11:57.000Z" }, { "content_html": "Comments", "url": "https://openstartups.listt.xyz", "title": "Startups that have hit more than $1K MRR in about 12-24 months", "date_modified": "2020-11-09T16:36:25.000Z" }, { "content_html": "Comments", "url": "https://nocomplexity.com/documents/arplaybook/index.html", "title": "Architecture Playbook", "date_modified": "2020-11-09T15:31:55.000Z" }, { "content_html": "Comments", "url": "https://postgresqlco.nf/en/doc/param/", "title": "PostgreSQL Configuration for Humans", "date_modified": "2020-11-08T08:52:16.000Z" }, { "content_html": "Comments", "url": "https://slavovojacek.medium.com/grpc-on-node-js-with-buf-and-typescript-part-1-5aad61bab03b", "title": "gRPC on Node.js with Buf and TypeScript", "date_modified": "2020-11-05T13:55:10.000Z" }, { "content_html": "\"Over\n\n Image: Cartoon Network\n\n

The tracks arrive on the 6th anniversary of the miniseries

\n

\n Continue reading…\n

", "url": "https://www.polygon.com/animation-cartoons/2020/11/3/21547498/over-the-garden-wall-new-songs", "title": "Over the Garden Wall composers release 6 new songs that were cut from the series", "date_modified": "2020-11-03T14:41:14.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/HammadH4/status/1323400864892022784", "title": "How not to SaaS – lessons from 2 years of building 3k MRR SaaS", "date_modified": "2020-11-02T23:07:33.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=24965115", "title": "Ask HN: How to effectively get feedback from users?", "date_modified": "2020-11-02T05:49:32.000Z" }, { "content_html": "Comments", "url": "https://vlfig.me/posts/microservices", "title": "Microservices – architecture nihilism in minimalism's clothes", "date_modified": "2020-11-02T00:16:23.000Z" }, { "content_html": "

Comments

", "url": "https://jjasghar.github.io/blog/2020/10/30/how-to-use-skopeo-to-migrate-off-dockerhub/", "title": "How to use skopeo to migrate off Docker Hub", "date_modified": "2020-10-31T17:28:18.000Z" }, { "content_html": "Comments", "url": "https://dayzero.substack.com/p/product-market-fit-for-a-b2b-company", "title": "What does product market fit look for a B2B startup?", "date_modified": "2020-10-31T15:33:09.000Z" }, { "content_html": "Comments", "url": "https://changelog.com/podcast/417#transcript", "title": "What's so about Postgres? (transcript)", "date_modified": "2020-10-30T17:43:50.000Z" }, { "content_html": "Comments", "url": "https://one.google.com/about/vpn", "title": "VPN by Google One", "date_modified": "2020-10-29T21:05:21.000Z" }, { "content_html": "Comments", "url": "https://krebsonsecurity.com/2020/10/fbi-dhs-hhs-warn-of-imminent-credible-ransomware-threat-against-u-s-hospitals/", "title": "FBI, DHS, HHS Warn of Imminent,Credible Ransomware Threat Against U.S. Hospitals", "date_modified": "2020-10-29T00:49:22.000Z" }, { "content_html": "

Comments

", "url": "https://designingfortheweb.co.uk/", "title": "Designing for the Web, by Mark Boulton", "date_modified": "2020-10-28T23:50:25.000Z" }, { "content_html": "Comments", "url": "https://www.richardesigns.co.uk/2020/10/27/what-was-remote-yc-like.html", "title": "What Was Remote YC Like?", "date_modified": "2020-10-28T08:47:36.000Z" }, { "content_html": "Comments", "url": "https://redditblog.com/2020/10/27/evolving-reddits-workforce/", "title": "Evolving Reddit’s Workforce", "date_modified": "2020-10-27T19:46:31.000Z" }, { "content_html": "Comments", "url": "https://github.com/rome/tools", "title": "The Rome Toolchain: A linter, compiler, bundler, and more", "date_modified": "2020-10-24T22:29:39.000Z" }, { "content_html": "Comments", "url": "https://alexdanco.com/2020/10/23/six-lessons-from-six-months-at-shopify/", "title": "Six Lessons from Six Months at Shopify", "date_modified": "2020-10-24T03:16:50.000Z" }, { "content_html": "Comments", "url": "https://github.com/github/dmca/blob/master/2020/10/2020-10-23-RIAA.md", "title": "YouTube-dl has received a DMCA takedown from RIAA", "date_modified": "2020-10-23T19:26:35.000Z" }, { "content_html": "Comments", "url": "https://pitch.com/blog/pitch-launches", "title": "Pitch Launches", "date_modified": "2020-10-20T12:49:54.000Z" }, { "content_html": "Comments", "url": "https://ianvanagas.com/2020/10/19/how-discord-won/", "title": "How Discord Won", "date_modified": "2020-10-19T18:56:53.000Z" }, { "content_html": "Comments", "url": "https://timhutton.github.io/GravityIsNotAForce/", "title": "Gravity is not a force – free-fall parabolas are straight lines in spacetime", "date_modified": "2020-10-18T21:07:26.000Z" }, { "content_html": "Comments", "url": "https://temporal.io/", "title": "Temporal: Build Invincible Apps", "date_modified": "2020-10-18T04:57:37.000Z" }, { "content_html": "Comments", "url": "https://keepthescore.co/blog/posts/deleting_the_production_database/", "title": "I deleted the production database by accident", "date_modified": "2020-10-17T22:09:05.000Z" }, { "content_html": "Comments", "url": "https://www.atlassian.com/migration/journey-to-cloud?jobid=104830907&subid=1515944789", "title": "Atlassian moves to cloud, going to stop selling server licenses", "date_modified": "2020-10-16T22:47:17.000Z" }, { "content_html": "Comments", "url": "https://www.hashicorp.com/blog/announcing-waypoint", "title": "HashiCorp Waypoint", "date_modified": "2020-10-15T16:02:56.000Z" }, { "content_html": "

Comments

", "url": "https://medium.com/faun/terraform-at-scale-modualized-hierachical-layout-cb5dbe5a368d", "title": "Terraform at Scale — Modualized Hierachical Layout", "date_modified": "2020-10-15T11:25:40.000Z" }, { "content_html": "Comments", "url": "https://www.percona.com/blog/2020/10/14/announcing-pg_stat_monitor-tech-preview-get-better-insights-into-query-performance-in-postgresql/", "title": "Pg_stat_monitor – better insights into query performance in PostgreSQL", "date_modified": "2020-10-14T15:29:38.000Z" }, { "content_html": "Comments", "url": "https://quaderno.io/blog/is-hacker-news-a-good-platform-to-launch-your-product/", "title": "Is Hacker News a Good Platform to Launch Your Product?", "date_modified": "2020-10-13T16:11:09.000Z" }, { "content_html": "Comments", "url": "https://medium.com/faun/terraform-at-scale-modualized-hierachical-layout-cb5dbe5a368d", "title": "Terraform at Scale", "date_modified": "2020-10-12T14:40:46.000Z" }, { "content_html": "Comments", "url": "https://blog.cloudflare.com/introducing-cloudflare-one/", "title": "Cloudflare One", "date_modified": "2020-10-12T13:01:59.000Z" }, { "content_html": "Comments", "url": "http://dtrace.org/blogs/bmc/2020/10/11/rust-after-the-honeymoon/", "title": "Rust After the Honeymoon", "date_modified": "2020-10-11T17:31:14.000Z" }, { "content_html": "Comments", "url": "https://www.cncf.io/announcements/2020/10/07/cloud-native-computing-foundation-announces-rook-graduation/", "title": "Cloud Native Computing Foundation Announces Rook Graduation", "date_modified": "2020-10-10T11:24:10.000Z" }, { "content_html": "Comments", "url": "https://www.forbes.com/sites/alexkonrad/2020/10/09/twilio-to-acquire-cloud-startup-segment-for-3-billion/", "title": "Twilio Set to Acquire Cloud Customer Data Startup Segment for $3.2B", "date_modified": "2020-10-09T23:23:15.000Z" }, { "content_html": "

Comments

", "url": "https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6283b0", "title": "Amazon's Build System", "date_modified": "2020-10-08T18:30:32.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=24719525", "title": "Ask HN: What are the pros / cons of using monorepos?", "date_modified": "2020-10-08T14:46:02.000Z" }, { "content_html": "Comments", "url": "https://research.securitum.com/mutation-xss-via-mathml-mutation-dompurify-2-0-17-bypass/", "title": "DOMPurify bypass: XSS via HTML namespace confusion", "date_modified": "2020-10-06T22:43:10.000Z" }, { "content_html": "

Comments

", "url": "https://pawelurbanek.com/heroku-postgres-aws-rds", "title": "Why You Should Migrate your Heroku Postgres Database to AWS RDS", "date_modified": "2020-10-06T08:21:20.000Z" }, { "content_html": "Comments", "url": "https://canny.io/blog/how-we-built-a-1m-arr-saas-startup/", "title": "How we built a $1m ARR SaaS startup", "date_modified": "2020-10-05T16:39:24.000Z" }, { "content_html": "Comments", "url": "https://www.btao.org/2020/10/02/npm-trust.html", "title": "A Web of Trust for NPM", "date_modified": "2020-10-03T18:51:42.000Z" }, { "content_html": "Comments", "url": "https://cra.mr/an-honest-review-of-gatsby/", "title": "An Honest Review of Gatsby", "date_modified": "2020-10-03T07:18:12.000Z" }, { "content_html": "Comments", "url": "https://www.iqt.org/bewear-python-typosquatting-is-about-more-than-typos/", "title": "The Danger of a Typo: An Investigation of Python and Typosquatting", "date_modified": "2020-10-01T11:35:39.000Z" }, { "content_html": "Comments", "url": "https://www.socialcooling.com/", "title": "Social Cooling", "date_modified": "2020-09-29T13:12:50.000Z" }, { "content_html": "Comments", "url": "https://www.mikesonders.com/saas-website-content/", "title": "The SaaS website content you need to close sales", "date_modified": "2020-09-28T20:56:36.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=24604687", "title": "Ask HN: Technical skills helpful for building a startup today", "date_modified": "2020-09-27T06:38:59.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=24601579", "title": "Ask HN: How to learn sales?", "date_modified": "2020-09-26T19:44:55.000Z" }, { "content_html": "Comments", "url": "https://baremetrics.com/blog/saas-financial-model", "title": "SaaS Financial Model", "date_modified": "2020-09-25T11:59:18.000Z" }, { "content_html": "Comments", "url": "https://zachdaniel.dev/learning-how-to-learn-japanese/", "title": "Learning How to Learn Japanese", "date_modified": "2020-09-22T18:15:03.000Z" }, { "content_html": "Comments", "url": "https://stripe.com/en-in/atlas/guides/saas-pricing", "title": "Pricing Low-Touch SaaS", "date_modified": "2020-09-21T13:52:43.000Z" }, { "content_html": "\n\n\n

When does a model own her own image?

\n

\n Continue reading…\n

", "url": "https://www.thecut.com/article/emily-ratajkowski-owning-my-image-essay.html", "title": "Buying myself back", "date_modified": "2020-09-15T21:48:22.000Z" }, { "content_html": "Comments", "url": "https://relinx.io/2020/09/14/old-good-database-design/", "title": "Old, Good Database Design", "date_modified": "2020-09-14T05:40:30.000Z" }, { "content_html": "Comments", "url": "https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/", "title": "How we upgraded PostgreSQL", "date_modified": "2020-09-12T16:51:54.000Z" }, { "content_html": "Comments", "url": "http://amyunger.com/blog/2020/09/10/staff-engineer-at-heroku.html", "title": "How I operated as a staff engineer at Heroku", "date_modified": "2020-09-10T23:32:34.000Z" }, { "content_html": "Comments", "url": "https://www.polarsparc.com/xhtml/OAuth2-OIDC.html", "title": "Understanding OAuth2 and OpenID Connect", "date_modified": "2020-09-06T14:37:09.000Z" }, { "content_html": "Comments", "url": "https://erickhun.com/posts/kubernetes-faster-services-no-cpu-limits/", "title": "Kubernetes: Make your services faster by removing CPU limits", "date_modified": "2020-09-02T09:47:30.000Z" }, { "content_html": "Comments", "url": "https://www.howmanydayssinceajwtalgnonevuln.com", "title": "# of days since the last alg=none JWT vulnerability", "date_modified": "2020-09-01T19:59:02.000Z" }, { "content_html": "Comments", "url": "https://blog.codonomics.com/2020/08/multi-tenant-architectures.html", "title": "Multi-Tenant Architectures", "date_modified": "2020-08-30T17:06:29.000Z" }, { "content_html": "Comments", "url": "https://tryhexadecimal.com/journal/business-taxes", "title": "First tax year with Stripe Atlas", "date_modified": "2020-08-26T14:46:29.000Z" }, { "content_html": "Comments", "url": "https://cloudflaremirrors.com/", "title": "Cloudflare Package Repository Mirrors", "date_modified": "2020-08-22T10:38:18.000Z" }, { "content_html": "Comments", "url": "https://www.cybertec-postgresql.com/en/recursive-queries-postgresql/", "title": "Understanding Recursive Queries in PostgreSQL", "date_modified": "2020-08-17T04:46:01.000Z" }, { "content_html": "Comments", "url": "https://abe-winter.github.io/2020/08/03/yr-of-git.html", "title": "One year of automatic DB migrations from Git", "date_modified": "2020-08-03T22:02:36.000Z" }, { "content_html": "Comments", "url": "https://heap.io/blog/engineering/postgresqls-powerful-new-join-type-lateral", "title": "PostgreSQL’s New Join Type: LATERAL (2014)", "date_modified": "2020-07-30T13:43:09.000Z" }, { "content_html": "

Comments

", "url": "https://mike.place/2020/resume.md/", "title": "resume.md: a Markdown resume", "date_modified": "2020-07-26T22:28:13.000Z" }, { "content_html": "Comments", "url": "https://github.com/hwayne/awesome-cold-showers", "title": "Cold Showers: For when people get too hyped up about things", "date_modified": "2020-07-24T05:08:17.000Z" }, { "content_html": "Comments", "url": "https://react-spectrum.adobe.com/react-spectrum/", "title": "A React implementation of Spectrum, Adobe’s design system", "date_modified": "2020-07-22T17:39:04.000Z" }, { "content_html": "Comments", "url": "https://uiplaybook.dev", "title": "Show HN: UI Playbook – A documented collection of UI components", "date_modified": "2020-07-22T13:42:11.000Z" }, { "content_html": "Comments", "url": "https://subvert.substack.com/p/stripe-building-a-developer-cult", "title": "Building a Developer Cult", "date_modified": "2020-07-21T02:19:25.000Z" }, { "content_html": "Comments", "url": "https://aws.amazon.com/blogs/containers/introducing-aws-copilot/", "title": "AWS Copilot", "date_modified": "2020-07-14T23:08:29.000Z" }, { "content_html": "Comments", "url": "https://arstechnica.com/gaming/2020/07/valve-secrets-spill-over-including-half-life-3-in-new-steam-documentary-app/", "title": "Valve secrets spill over, including Half-Life 3, in new Steam documentary app", "date_modified": "2020-07-10T03:57:04.000Z" }, { "content_html": "Comments", "url": "https://tyrrrz.me/blog/unit-testing-is-overrated", "title": "Unit Testing Is Overrated", "date_modified": "2020-07-09T11:06:40.000Z" }, { "content_html": "Comments", "url": "https://www.youtube.com/watch?v=g_ZdcHSFGv0&t=10s", "title": "Professor solves 240 computer science exam problems in 4 hours", "date_modified": "2020-07-07T14:34:03.000Z" }, { "content_html": "Comments", "url": "http://linear.app/", "title": "Linear – A fast issue tracker", "date_modified": "2020-06-30T18:16:16.000Z" }, { "content_html": "Comments", "url": "https://static.sched.com/hosted_files/ossna2020/fc/Running%20Postgres-as-a-Service%20in%20Kubernetes.pdf", "title": "Running Postgres in Kubernetes [pdf]", "date_modified": "2020-06-29T20:29:12.000Z" }, { "content_html": "Comments", "url": "https://sambleckley.com/writing/npm.html", "title": "Worrying about the NPM Ecosystem", "date_modified": "2020-06-29T15:21:31.000Z" }, { "content_html": "Comments", "url": "https://medium.com/@melissamcewen/a-guide-to-hacker-news-for-people-who-arent-men-5737bc3e68a", "title": "A Guide to Hacker News for People Who Aren’t Men – Melissa Mcewen", "date_modified": "2020-06-29T14:09:06.000Z" }, { "content_html": "Comments", "url": "https://blog.fleetsmith.com/fleetsmith-acquired-by-apple/", "title": "Apple Acquires Fleetsmith", "date_modified": "2020-06-24T15:55:18.000Z" }, { "content_html": "Comments", "url": "https://softwareengineeringdaily.com/2020/02/13/setting-the-stage-for-platform-engineering/", "title": "The Rise of Platform Engineering", "date_modified": "2020-06-21T12:07:41.000Z" }, { "content_html": "Comments", "url": "http://www.mpe.mpg.de/7461761/news20200619", "title": "Our deepest view of the X-ray sky", "date_modified": "2020-06-19T12:00:59.000Z" }, { "content_html": "Comments", "url": "https://github.blog/2020-06-18-introducing-github-super-linter-one-linter-to-rule-them-all/", "title": "GitHub Super Linter: one linter to rule them all", "date_modified": "2020-06-18T15:14:40.000Z" }, { "content_html": "

It was May 2015, and the Hotel Surya in Varanasi, India, was hosting “Inner Awakening,” the flagship spiritual training program of Paramahamsa Nithyananda. To his followers, he is a god incarnate, “His Divine Holiness Bhagavan Sri Nithyananda Paramashivam,” the living avatar of Shiva, or, as one devotee put it, “the…

Read more...

", "url": "https://gizmodo.com/she-built-a-shady-guru-s-youtube-army-now-she-s-his-fi-1843283794", "title": "She Built a Shady Guru’s YouTube Army. Now She’s His Fiercest Critic—But Who Will Believe Her?", "date_modified": "2020-06-17T14:30:00.000Z" }, { "content_html": "Comments", "url": "https://jeffreymoro.com/blog/2020-02-13-against-cop-shit/", "title": "Against Cop Shit", "date_modified": "2020-06-16T17:42:54.000Z" }, { "content_html": "Comments", "url": "https://www.vice.com/en_us/article/v7gd9b/facebook-helped-fbi-hack-child-predator-buster-hernandez", "title": "Facebook Helped Develop a Tails Exploit", "date_modified": "2020-06-16T06:19:36.000Z" }, { "content_html": "Comments", "url": "https://docs.google.com/spreadsheets/u/1/d/1YmZeSxpz52qT-10tkCjWOwOGkQqle7Wd1P7ZM1wMW0E/htmlview?pru=AAABcql6DI8*mIHYeMnoj9XWUp3Svb_KZA", "title": "George Floyd Protest – police brutality videos on Twitter", "date_modified": "2020-06-15T00:12:40.000Z" }, { "content_html": "

Author: Zach Corleissen, Cloud Native Computing Foundation

\n

Editor's note: Zach is one of the chairs for the Kubernetes documentation special interest group (SIG Docs).

\n

I'm pleased to announce that the Kubernetes website now features the Docsy Hugo theme.

\n

The Docsy theme improves the site's organization and navigability, and opens a path to improved API references. After over 4 years with few meaningful UX improvements, Docsy implements some best practices for technical content. The theme makes the Kubernetes site easier to read and makes individual pages easier to navigate. It gives the site a much-needed facelift.

\n

For example: adding a right-hand rail for navigating topics on the page. No more scrolling up to navigate!

\n

The theme opens a path for future improvements to the website. The Docsy functionality I'm most excited about is the theme's swaggerui shortcode, which provides native support for generating API references from an OpenAPI spec. The CNCF is partnering with Google Season of Docs (GSoD) for staffing to make better API references a reality in Q4 this year. We're hopeful to be chosen, and we're looking forward to Google's list of announced projects on August 16th. Better API references have been a personal goal since I first started working with SIG Docs in 2017. It's exciting to see the goal within reach.

\n

One of SIG Docs' tech leads, Karen Bradshaw did a lot of heavy lifting to fix a wide range of site compatibility issues, including a fix to the last of our legacy pieces when we migrated from Jekyll to Hugo in 2018. Our other tech leads, Tim Bannister and Taylor Dolezal provided extensive reviews.

\n

Thanks also to Björn-Erik Pedersen, who provided invaluable advice about how to navigate a Hugo upgrade beyond version 0.60.0.

\n

The CNCF contracted with Gearbox in Victoria, BC to apply the theme to the site. Thanks to Aidan, Troy, and the rest of the team for all their work!

", "url": "https://kubernetes.io/blog/2020/06/better-docs-ux-with-docsy/", "title": "Blog: A Better Docs UX With Docsy", "date_modified": "2020-06-15T00:00:00.000Z" }, { "content_html": "Comments", "url": "https://edition.cnn.com/2020/06/13/media/seattle-fox-news-autonomous-zone-protest/index.html", "title": "Fox News publishes altered and misleading images of Seattle demonstrations", "date_modified": "2020-06-14T07:38:27.000Z" }, { "content_html": "Comments", "url": "https://health.ucdavis.edu/health-news/contenthub/autism-severity-can-change-substantially-during-early-childhood/2020/05", "title": "Autism severity can change substantially during early childhood, study suggests", "date_modified": "2020-05-31T16:54:06.000Z" }, { "content_html": "Comments", "url": "https://www.michael-noll.com/blog/2020/01/16/what-every-software-engineer-should-know-about-apache-kafka-fundamentals/", "title": "What every software engineer should know about Apache Kafka", "date_modified": "2020-05-16T19:40:58.000Z" }, { "content_html": "Comments", "url": "https://diagrams.mingrammer.com/", "title": "Diagram as Code", "date_modified": "2020-05-12T15:13:46.000Z" }, { "content_html": "Comments", "url": "https://www.nytimes.com/2020/03/03/magazine/hideo-kojima-death-stranding-video-game.html", "title": "Hideo Kojima’s Strange, Unforgettable Video-Game Worlds", "date_modified": "2020-05-03T05:05:11.000Z" }, { "content_html": "Comments", "url": "https://layerci.com", "title": "Show HN: Layer – Get a dozen staging servers per developer", "date_modified": "2020-04-30T19:31:24.000Z" }, { "content_html": "Comments", "url": "https://www.talentica.com/blogs/distributed-transactions-are-not-micro-services/", "title": "Distributed transactions are not microservices", "date_modified": "2020-04-30T15:57:27.000Z" }, { "content_html": "Comments", "url": "https://medium.com/@rakyll/things-i-wished-more-developers-knew-about-databases-2d0178464f78", "title": "Things I Wished More Developers Knew About Databases", "date_modified": "2020-04-22T04:45:04.000Z" }, { "content_html": "Comments", "url": "https://nipponcolors.com", "title": "Nippon Colors", "date_modified": "2020-04-07T12:25:31.000Z" }, { "content_html": "Comments", "url": "https://gds.blog.gov.uk/2020/04/03/how-gov-uk-notify-reliably-sends-text-messages-to-users/", "title": "How GOV.UK Notify reliably sends text messages to users", "date_modified": "2020-04-04T14:11:38.000Z" }, { "content_html": "Comments", "url": "https://medium.com/@rbranson/10-things-i-hate-about-postgresql-20dbab8c2791", "title": "PostgreSQL's Imperfections", "date_modified": "2020-04-04T00:57:41.000Z" }, { "content_html": "Comments", "url": "https://github.com/mbrt/gmailctl", "title": "Gmailctl – Declarative Configuration for Gmail Filters", "date_modified": "2020-03-29T02:56:12.000Z" }, { "content_html": "
\"Meta

Meta for Mac.

\n

For the past year, I’ve been using a high-res Sony music player to listen to my personal music collection. I detailed the entire story in the December 2019 episode of our Club-exclusive MacStories Unplugged podcast, but in short: I still use Apple Music to stream music every day and discover new artists; however, for those times when I want to more intentionally listen to music without doing anything else, I like to sit down, put on my good Sony headphones, and try to enjoy all the sonic details of my favorite songs that wouldn’t normally be revealed by AirPods or my iPad Pro’s speakers. But this post isn’t about how I’ve been dipping my toes into the wild world of audiophiles and high-resolution music; rather, I want to highlight an excellent Mac app I’ve been using to organize and edit the metadata of the FLAC music library I’ve been assembling over the past year.

\n

\n

These days, when I think of an old album I want to repurchase in high resolution (either 16-bit or 24-bit FLAC), or if I come across a new release I instantly fall in love with, I go ahead and buy it as a standalone FLAC digital download.1 I then organize albums with a standard Artist ⇾ Album folder hierarchy in the Mac’s Finder, as pictured below:

\n
\"My

My FLAC music collection.

\n

Before you ask: yes, I could do this file organization with my iPad Pro alone because the Sony music player I use (this Walkman model) can be connected via USB to the iPad (with this adapter) and comes with a standard SD card for expandable storage. However, I prefer to purchase and download FLAC music on my Mac mini because my music collection is also backed up and mirrored to Plex, and the Mac mini – as you might imagine – is running a Plex media server instance in the background at all times. The music library is stored on a 1 TB Samsung T5 external SSD that’s connected via USB-C to the Mac mini; whenever I purchase new music, I manually copy it into the T5 as well as the Sony Walkman’s SD card via the Finder.

\n

Most of the time, FLAC music I purchase online comes with correct built-in metadata for fields such as track number, year, disc, and album artwork. But sometimes it doesn’t, which leads to the unfortunate situation of ending up with songs on my Walkman that lack album artwork or feature extra text in their titles such as “Remastered” or “Explicit”. It’s particularly annoying when artwork is missing because it ruins the experience of looking at the now playing screen while I’m focused on enjoying music. Plex doesn’t have this issue: by virtue of being an online service, Plex can search various databases for correct metadata and automatically fill missing fields in my library. One of the reasons I enjoy listening to my personal music library the old-school way is that the Walkman is completely offline, but that comes with the disadvantage of being unable to fix incorrect metadata via the web.

\n
\"An

An example of songs with incorrect metadata on my Walkman. The word \"Remastered\" shouldn't be part of the song title field.

\n
\"No

No album artwork makes me sad.

\n

This is where Meta, an advanced music tag editor for Mac developed by French indie developer Benjamin Jaeger, comes in. I came across Meta a few months ago when, frustrated with the ugliness and bloated nature of other desktop metadata editors, I took it upon myself to find a polished, modern tag editor designed specifically with Mac users in mind. Meta is exactly what I was looking for: the app is a modern Mac utility that supports all popular audio formats (from standard MP3 and MP4 to FLAC, DSF, and AIFF) and can write metadata formats such as ID3v1, ID3v2, MP4, and APE tags. Unlike other cross-platform or open-source tag editors, Meta’s feature set is focused on one area – editing metadata for your digital music collection – and supports all the common interface paradigms you’d expect from a professional Mac app running on Catalina.

\n

Meta can do a lot of different things, and this is by no means a comprehensive review of the app: after all, my needs are fairly basic – I drop an album into the app, fix the tags that are incorrect, and close the app. The Meta website has details and screenshots that cover all of the app’s features, and there’s also a full 3-day free trial you can use without limitations to take Meta for a spin. Having used the app for a while, I would describe it as follows: Meta is based on a powerful tag engine (called TagLib) and customizable, native Mac UI that supports a resizable sidebar, popovers, dark mode, keyboard shortcuts, and multiple view options; in addition to this flexible UI, Meta sports an incredible text-matching engine that lets you perform tasks such as batch renaming multiple files, creating new tags by mixing text and metadata tokens, finding and replacing text, and even converting file paths or metadata to new tags using regex patterns.

\n

As I noted above, that’s a lot to take in, so let me walk through my typical use of Meta. Once I’ve identified an album that needs some of its metadata fixed, I drop the entire folder from the Finder into Meta. If I want to add artwork to it, all I need to do is select all songs in Meta’s main list, then drag the artwork image I previously downloaded from Google Images into the artwork section of Meta’s sidebar. The artwork tag will be instantly written to all FLAC files at once, and that’s it.

\n
\"You

You can drag and drop album covers from the Finder.

\n

But what if you don’t want to manually find and download a high-quality album artwork image beforehand? Meta has you covered there as well: by purchasing the additional, one-time-only Cover Finder feature, you’ll be able to select multiple tracks and hit ⌘⇧K to let Meta find artwork images for you. In my experience, results provided by Meta have been perfect (images fetched by Meta are high-definition covers with the correct color saturation and no watermarks); at least for me, they’re worth the extra purchase so I don’t have to manually search for each album artwork and clutter my desktop with (often low-res) images downloaded from random websites found via Google.

\n
\"Meta

Meta can find album covers for you by searching online databases.

\n

Besides album artwork, I’ve also been using Meta to fix text-based metadata by taking advantage of the app’s powerful text engine. I often come across remastered editions of albums where each song has the word “Remastered” included in the title field of its metadata – e.g. instead of “Wonderwall”, the song is called “Wonderwall Remastered”. Meta makes these kinds of batch edits extremely easy: after selecting all songs, I can select Edit ⇾ Find ⇾ Find & Replace from the menu bar (or hit ⌥⌘F) to activate the app’s find and replace UI. From there, I can type the text I want to get rid of in the ‘Find’ field and select the ‘All’ button next to ‘Replace’ (a common UI element in Mac apps) to clear the unwanted text from all title fields at once. In a nice touch, Meta visually confirms text matches found in the selected items and even lets you add special tokens such as tab characters or white space if you want to further refine your searches.

\n
\"Meta's

Meta's find and replace UI.

\n
\"You

You can insert special characters in the find and replace UI with this menu.

\n

I could also edit each song’s metadata manually by clicking into the appropriate field in the sidebar (also pictured above), but for these batch operations, Meta’s find and replace system is ideal. If you don’t want to replace an existing tag’s value but compose a new one altogether, Meta can do that as well: after selecting an item (or multiple ones), hit the pencil button in the toolbar to open the ‘Compose Tag’ popup, which lets you use arbitrary plain text as well as built-in tokens to overwrite any existing tag value. In the screenshot below, you can see how I edited the ‘Year’ field for all songs contained in The Cure’s Greatest Hits album by manually typing “2001” into the Compose Tag box.

\n
\"Composing

Composing a metadata tag in Meta.

\n

Meta can also rename files in the Finder: hit ⌘⇧R and, as with tags, you can mix and match existing tags and plain text to rename files however you see fit. In the example below, I used the track number tag, a hyphen, and the title tag to come up with a file name pattern I like to see in the Finder. Meta even remembers your preferences for file renaming, so once you’ve found a style you like, you won’t have to recreate the naming pattern every single time.

\n
\"You

You can also let Meta rename files for you.

\n

There are dozens of advanced features in Meta I haven’t used myself, but which contribute to making this app the premier utility for managing music collections. In the app’s preferences, you can set different options for album artwork including format, compression, and max size; you can let the app move and organize files in subfolders for you using the ‘Create Directory’ feature, which, again, uses patterns to let you craft a custom file path; there’s support for editing built-in lyrics, dates (with a visual date picker), and even track number sequences. The only thing Meta can’t do for me is add lyrics in the LRC format Sony uses for the Walkman, but I can’t blame the developer for not supporting this particular aspect of Sony’s music ecosystem.

\n
\"Meta

Meta offers several customization options.

\n

The greatest compliment I can pay to Meta is that, despite its abundance of features, it never feels overwhelming, and every functionality is always where I expect it to be. Thanks to Meta, I’ve been able to clean up and reorganize my FLAC music collection so that every album now has correct metadata on my Walkman, and I’m happier because of it.

\n

Meta is a remarkable example of the kind of thoughtful, powerful professional tools indie developers can still create on macOS these days, and it’s one of my favorite Mac app discoveries in a long time. If you also like to manage your music library the old-fashioned way and can’t stand incorrect metadata, I can’t recommend Meta enough.

\n
\n
  1. \nWebsites I’ve been using to purchase FLAC music include 7digital, HDtracks, Qobuz, and some artists’ official Bandcamp profiles. ↩︎\n
    2. \n
\n

Support MacStories Directly

Club MacStories offers exclusive access to extra MacStories content, delivered every week; it’s also a way to support us directly.

\n

Club MacStories will help you discover the best apps for your devices and get the most out of your iPhone, iPad, and Mac. Plus, it’s made in Italy.

\n

Join Now", "url": "https://www.macstories.net/mac/editing-flac-metadata-with-meta-for-mac/", "title": "Editing FLAC Metadata with Meta for Mac", "date_modified": "2020-03-19T16:24:30.000Z" }, { "content_html": "Comments", "url": "https://capiche.com/ama/patrick-mckenzie-stripe", "title": "Ask Patrick McKenzie Anything", "date_modified": "2020-03-10T23:46:39.000Z" }, { "content_html": "Comments", "url": "https://github.com/calebj0seph/spectro/blob/master/docs/making-of.md", "title": "Building Spectro: a Real-Time WebGL audio spectrogram visualizer", "date_modified": "2020-03-06T17:10:03.000Z" }, { "content_html": "Comments", "url": "http://radiooooo.com/", "title": "Radiooooo – The Music Time Machine", "date_modified": "2020-03-05T07:20:45.000Z" }, { "content_html": "Comments", "url": "https://poolside.fm/", "title": "Poolside.fm", "date_modified": "2020-02-20T00:22:30.000Z" }, { "content_html": "

Comments

", "url": "https://blog.soykaf.com/post/postgresl-front-report/", "title": "Postgresql Search: From the trenches", "date_modified": "2020-02-18T15:01:43.000Z" }, { "content_html": "Comments", "url": "https://www.mint-lang.com/", "title": "Mint: The programming language for writing single page applications", "date_modified": "2020-02-05T13:44:06.000Z" }, { "content_html": "Comments", "url": "https://blog.wesleyac.com/posts/engineering-beliefs", "title": "Things I Believe About Software Engineering", "date_modified": "2020-02-03T07:28:56.000Z" }, { "content_html": "

Comments

", "url": "https://blog.wesleyac.com/posts/engineering-beliefs", "title": "Things I Believe About Software Engineering", "date_modified": "2020-02-02T17:12:01.000Z" }, { "content_html": "Comments", "url": "https://changelog.com/posts/monoliths-are-the-future", "title": "Monoliths Are the Future", "date_modified": "2020-01-30T17:41:50.000Z" }, { "content_html": "Comments", "url": "https://github.com/typesense/typesense", "title": "Typesense: Open-Source Alternative to Algolia", "date_modified": "2020-01-29T15:20:39.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=22157166", "title": "Ask HN: How do you currently solve authentication?", "date_modified": "2020-01-27T06:06:12.000Z" }, { "content_html": "Comments", "url": "https://www.unisonweb.org/docs/tour/", "title": "Unison: A Content-Addressable Programming Language", "date_modified": "2020-01-27T02:16:28.000Z" }, { "content_html": "

Comments

", "url": "https://christine.website/blog/dhall-kubernetes-2020-01-25", "title": "Dhall for Kubernetes", "date_modified": "2020-01-25T21:26:45.000Z" }, { "content_html": "Comments", "url": "https://danluu.com/wat/", "title": "Normalization of Deviance (2015)", "date_modified": "2020-01-25T02:58:06.000Z" }, { "content_html": "Comments", "url": "https://www.ongres.com/blog/postgresqlconf-configuration-for-humans/", "title": "Show HN: Postgresqlco.nf: PostgreSQL Configuration for Humans", "date_modified": "2020-01-24T16:51:16.000Z" }, { "content_html": "Comments", "url": "https://www.ncsc.gov.uk/whitepaper/security-architecture-anti-patterns", "title": "Security Architecture Anti-Patterns", "date_modified": "2020-01-15T14:48:40.000Z" }, { "content_html": "Comments", "url": "https://www.jetbrains.com/lp/mono/", "title": "JetBrains Mono: A free and open-source typeface for developers", "date_modified": "2020-01-15T13:39:50.000Z" }, { "content_html": "Comments", "url": "https://cassandraxia.com/writing/police.html", "title": "What to do if you’re stopped by the police", "date_modified": "2020-01-14T00:25:21.000Z" }, { "content_html": "Comments", "url": "https://www.issms2fasecure.com/", "title": "Is SMS 2FA Secure?", "date_modified": "2020-01-10T22:16:24.000Z" }, { "content_html": "Comments", "url": "http://www.overcomingbias.com/2020/01/how-bees-argue.html", "title": "Bees Argue", "date_modified": "2020-01-08T06:37:10.000Z" }, { "content_html": "Comments", "url": "https://www.nytimes.com/interactive/2019/12/19/opinion/location-tracking-cell-phone.html", "title": "An investigation into the smartphone tracking industry", "date_modified": "2019-12-19T10:46:37.000Z" }, { "content_html": "Comments", "url": "https://www.espn.com/esports/story/_/id/28319463/league-legends-quickly-gaining-traditional-sports-american-popularity", "title": "League of Legends quickly gaining on traditional sports in American popularity", "date_modified": "2019-12-19T03:21:52.000Z" }, { "content_html": "Comments", "url": "https://blog.eventide-project.org/articles/announcing-message-db/", "title": "Message DB: Event Store and Message Store for PostgreSQL", "date_modified": "2019-12-17T03:33:40.000Z" }, { "content_html": "Comments", "url": "https://www.inklestudios.com/ink/web-tutorial/", "title": "Writing web-based interactive fiction with Ink", "date_modified": "2019-12-15T07:29:02.000Z" }, { "content_html": "Comments", "url": "http://www.aidungeon.io/", "title": "Show HN: AI Dungeon 2, An AI generated text adventure built 1.5B param GPT-2", "date_modified": "2019-12-05T21:54:36.000Z" }, { "content_html": "Comments", "url": "https://twitter.com/patio11/status/1201003855770607618", "title": "Shibboleths that get you past the initial script stage", "date_modified": "2019-12-01T08:51:53.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=21656891", "title": "Ask HN: How do you deal with atomicity in microservice environments?", "date_modified": "2019-11-28T12:48:40.000Z" }, { "content_html": "Comments", "url": "https://www.pika.dev/registry", "title": "Pika – A JavaScript package registry for the modern web", "date_modified": "2019-11-24T21:00:18.000Z" }, { "content_html": "Comments", "url": "https://half-life.com/en/alyx", "title": "Half-Life: Alyx", "date_modified": "2019-11-21T18:31:16.000Z" }, { "content_html": "Comments", "url": "https://www.eurogamer.net/articles/2019-08-06-the-creepy-corridors-of-video-games", "title": "The Creepy Corridors of Video Games", "date_modified": "2019-11-10T19:48:22.000Z" }, { "content_html": "Comments", "url": "https://monzo.com/blog/we-built-network-isolation-for-1-500-services", "title": "We built network isolation for 1500 services", "date_modified": "2019-11-05T14:31:30.000Z" }, { "content_html": "Comments", "url": "https://paulstamatiou.com/getting-started-with-security-keys/", "title": "Getting Started with Security Keys", "date_modified": "2019-10-31T01:00:08.000Z" }, { "content_html": "Comments", "url": "https://backgroundchecks.org/justdeleteme/", "title": "Just Delete Me – A directory of direct links to delete your account", "date_modified": "2019-09-02T14:14:55.000Z" }, { "content_html": "Comments", "url": "http://japanesecomplete.com/777", "title": "Most Frequent 777 Kanji Give You 90.0% Coverage of Kanji in the Wild", "date_modified": "2019-08-17T03:12:06.000Z" }, { "content_html": "Comments", "url": "https://github.com/hjacobs/kubernetes-failure-stories", "title": "Kubernetes Failure Stories", "date_modified": "2019-06-12T11:32:32.000Z" }, { "content_html": "Comments", "url": "https://www.newyorker.com/magazine/2019/05/06/my-childhood-in-a-cult", "title": "My Childhood in a Cult", "date_modified": "2019-05-02T21:45:32.000Z" }, { "content_html": "Comments", "url": "http://www.engineerbetter.com/blog/yubikey-all-the-things/", "title": "Show HN: Yubikey guide for Git Signing, SSH Auth, U2F 2FA, and 1Password", "date_modified": "2019-04-30T12:12:22.000Z" }, { "content_html": "Comments", "url": "https://bittersoutherner.com/waffle-house-vistas", "title": "Waffle House Vistas", "date_modified": "2019-03-15T18:07:38.000Z" }, { "content_html": "Comments", "url": "https://www.nytimes.com/interactive/2019/02/21/magazine/autism-office-design.html", "title": "An Office Designed for Workers with Autism", "date_modified": "2019-02-22T11:44:38.000Z" }, { "content_html": "Comments", "url": "https://www.thedailybeast.com/inside-the-secret-facebook-war-for-mormon-hearts-and-minds?via=twitter_page", "title": "The Secret Facebook War for Mormon Hearts and Minds", "date_modified": "2019-02-10T23:15:21.000Z" }, { "content_html": "Comments", "url": "http://www.nobadmemories.com/blog/2017/04/better-together-displaying-japanese-and-english-text-on-the-web/", "title": "Displaying Japanese and English Text on the Web", "date_modified": "2019-02-08T11:06:05.000Z" }, { "content_html": "Comments", "url": "https://www.bbc.com/news/av/stories-46885707/rent-a-sister-coaxing-japan-s-young-men-out-of-their-rooms", "title": "Rent-a-sister: Coaxing Japan’s young men out of their rooms [video]", "date_modified": "2019-01-20T14:54:41.000Z" }, { "content_html": "
\n
\"A

Enlarge (credit: Getty | Photofusion)

\n

The American Psychological Association is on the defensive over its newly released clinical guidance (PDF) for treating boys and men, which links traditional masculinity ideology to a range of harms, including sexism, violence, mental health issues, suicide, and homophobia. Critics contend that the guidelines attack traditional values and innate characteristics of males.

\n

The APA’s 10-point guidance, released last week, is intended to help practicing psychologists address the varied yet gendered experience of men and boys with whom they work. It fits into the APA’s set of other clinical guidelines for working with specific groups, including older adults, people with disabilities, and one for girls and women, which was released in 2007. The association began working on the guidance for boys and men in 2005—well before the current #MeToo era—and drew from more than four decades of research for its framing and recommendations.

\n

That research showed that “some masculine social norms can have negative consequences for the health of boys and men,” the APA said in a statement released January 14 amid backlash. Key among these harmful norms is pressure for boys to suppress their emotions (the “common ‘boys don’t cry’ refrain”), the APA said. This has been documented to lead to “increased negative risk-taking and inappropriate aggression among men and boys, factors that can put some males at greater risk for psychological and physical health problems.” It can also make males “less willing to seek help for psychological distress.”

\n

Read 8 remaining paragraphs | Comments

\n \n
", "url": "https://arstechnica.com/?p=1441635", "title": "Psychologists defend claim of “destructive aspects” to masculinity", "date_modified": "2019-01-15T22:20:25.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=18770366", "title": "Ask HN: How do you store photos and videos?", "date_modified": "2018-12-27T16:19:31.000Z" }, { "content_html": "Comments", "url": "https://www.ucsf.edu/news/2018/12/412916/sugars-sick-secrets-how-industry-forces-have-manipulated-science-downplay-harm", "title": "Sugar’s Sick Secrets: How Industry Forces Manipulated Science to Downplay Harm", "date_modified": "2018-12-26T22:40:56.000Z" }, { "content_html": "Comments", "url": "https://optician-sans.com/", "title": "Optician Sans – A free font based on the historical eye charts", "date_modified": "2018-12-12T17:08:37.000Z" }, { "content_html": "

Fallout: New California, an enormously ambitious mod for Fallout: New Vegas, is now out, finished and ready to play.

New California—built from the bones of an older mod called Brazil— is set in between the events of Fallout 2 and Fallout New Vegas, and to simply call it a “mod” is to sell it short. This is practically a brand new, fan-made Fallout game, that’s even got voice acting and takes place on a new map, with the game set in the Black Bear Mountain National Forest in California.

Advertisement

Its release is timed well; Fallout 76 is out soon, and anyone disappointed that it’s a multiplayer affair, and not the traditional epic singleplayer RPG, can just try this out instead.

You can download the mod here. As for how good it is, Nathan is playing it right now, and will have some impressions up on Kotaku soon!

Advertisement

UPDATE: This post’s headline earlier referred to the mod being nine years old. It’s actually been in active development since 2012.

", "url": "https://kotaku.com/massive-fallout-mod-nine-years-in-the-making-is-out-tod-1829949486", "title": "Massive Fallout Mod New California Is Out Today", "date_modified": "2018-10-23T23:00:00.000Z" }, { "content_html": "Comments", "url": "https://github.com/rtr7/router7", "title": "Show HN: Router7 – A pure-Go implementation of a small home internet router", "date_modified": "2018-07-14T12:35:16.000Z" }, { "content_html": "Comments", "url": "https://www.newyorker.com/culture/culture-desk/the-united-states-of-japan", "title": "The United States of Japan", "date_modified": "2018-05-05T00:03:37.000Z" }, { "content_html": "

The Environmental Working Group has released their latest “Dirty Dozen” list of supposedly pesticide-laden fruits and vegetables. (This is a misleading list, as we’ve explained before.) You may be tempted to buy organic produce, as the EWG suggests, but guess what—organic produce is not pesticide-free.

Organic farmers may use pesticides, so long as they choose from a list of approved options. The USDA organic program does not disallow all pesticides, just “synthetic” ones. (By the way, the term “pesticides” includes both bug sprays and weed killers.)

Advertisement

So what remains on our vegetables? The USDA periodically tests produce for pesticide residues; this is the Pesticide Data Program. (The EWG repurposes this data to create their Dirty Dozen and Clean Fifteen lists.) But the USDA does not test for the presence of organic-allowed pesticides. So the EWG is reporting the stuff on conventional crops without considering what’s present on organic crops.

So, will you lower your pesticide exposure by switching to organic? We don’t know, but the answer may very well be no. Even looking at the synthetic, non-organic pesticides in the USDA’s tests, conventional crops don’t always have the lowest amounts. Take strawberries, for example, the “dirtiest” item on the 2018 list: 75 percent of organic strawberries, and 76 percent of conventional strawberries, had pesticide levels that were under 5 percent of the allowable levels.

In other words, buying organic strawberries might expose you to more pesticide residues than buying conventional. We recommend ignoring the Dirty Dozen list entirely, and buying whichever fruits and veggies work for your diet and your budget.

", "url": "https://vitals.lifehacker.com/organic-food-has-pesticides-too-1825156951", "title": "Organic Food Has Pesticides, Too", "date_modified": "2018-04-10T21:30:00.000Z" }, { "content_html": "Comments", "url": "https://www.theguardian.com/world/2018/mar/21/christianity-non-christian-europe-young-people-survey-religion", "title": "'Christianity as default is gone': the rise of a non-Christian Europe", "date_modified": "2018-03-21T15:03:42.000Z" }, { "content_html": "\n\n\n\n

Adult Swim’s hit animated show Rick and Morty finished its third season back in October, and we might not get season 4 until 2019, according to one of the show’s writers. To tide us over, Adult Swim just released a music video for Run The Jewels’ song “Oh Mama,” featuring the mad scientist and his grandson.

\n

The video is directed by Rick and Morty director Juan Meza-León (he was responsible for episodes like “The Rickshank Rickdemption,” “The Whirly Dirly Conspiracy,” and “The ABC’s of Beth”), and it plays out a bit like a regular episode of the show: Rick and Morty fly off to a random planet, crash a club full of insectoid Gromflamites, and carnage ensues. They steal a briefcase and head out, only to encounter some additional problems,...

\n

\n Continue reading…\n

", "url": "https://www.theverge.com/2018/3/16/17130166/rick-and-morty-run-the-jewels-oh-mama-music-video-watch", "title": "Adult Swim’s new music video for Run the Jewels’ Oh Mama is like a mini Rick and Morty episode", "date_modified": "2018-03-16T16:53:26.000Z" }, { "content_html": "
Nintendo’s original headquarters in Shimokyo-ku, Kyoto, Japan, in 1889, the year of the company’s founding. Photo: City of Kyoto

It’s well-known that Nintendo was originally founded in Kyoto, Japan as a maker of playing cards in 1889. But a recent historical project by the city of Kyoto has turned up, for the first time, a photograph of what the company’s headquarters looked like in that year.

The photo, and an accompanying blog post, were published in December on “Memories of Kyoto, 150 Years After The Meiji Period,” an ongoing historical project documenting the city’s history during the reign of Emperor Meiji from 1868 to 1912. Nintendo historians authors Florent Gorges and Isao Yamazaki shared them around today, noting that the blog post was full of fascinating little-known information about Nintendo’s founding.

Advertisement

Nintendo’s founder Fusajiro Yamauchi originally ran a company called Haiko, which at the time specialized in cement, the post says. His name was originally Fusajiro Fukui, but he was adopted as an adult by his boss Naoshichi Yamauchi. This is actually extremely common in Japan—government records show that the vast majority of adoptions are adult men adopting other adult men, so that their companies can remain a “family business” even when there is no biological son to inherit the company.

Nintendo stayed a family business until the retirement of Fusajiro’s great-grandson Hiroshi Yamauchi in 2002, after which Satoru Iwata took the helm. Haiko is still in operation and is run by Kazumasa Yamauchi, who wrote the City of Kyoto’s blog post.

In 1889, Fusajiro Yamauchi struck out on his own to form Marufuku Nintendo Card Co., producing traditional Japanese playing cards called hanafuda and eventually Western playing cards as well. Here’s a map of where that original headquarters used to be located, in case you want to make a pilgrimage.

If you want to see the oldest Nintendo building that’s still actually standing, there’s one very close to downtown Kyoto. According to Gorges and Yamazaki’s book, the building that housed the original headquarters was demolished in 2004, and replaced with a parking lot.

", "url": "https://kotaku.com/heres-what-nintendos-hq-looked-like-in-1889-1822338950", "title": "Here's What Nintendo's HQ Looked Like In 1889", "date_modified": "2018-01-23T18:00:00.000Z" }, { "content_html": "

The Red Strings Club starts, and ends, with a character falling out of a high-rise window. There’s no indication that you could do anything to stop this from happening, or even that you’re supposed to. The game is more interested in how you get to that point—and the web of lies, manipulation, and tough choices you leave in your wake.

This cyberpunk point-and-click game, out today on PC, is by Deconstructeam, the developers behind 2014’s Gods Will Be Watching. Gods was a series of scenarios in which the player had to manage characters’ needs and unclear emotional states. The Red Strings Club is set up roughly the same way, but it’s not the brutal tightrope walk of life and death its predecessor was. It’s more forgiving, but it’s also more complicated. Its decisions open up into possibilities, nuances, and outcomes that don’t have clear rights and wrongs.

The game mostly takes place in the titular Red Strings Club, a bar in a cyberpunk dystopia owned by a man named Donovan. In this future, people have cybernetic implants, but Donovan has a medical condition that prevents him from getting them. This gives him a fairly anti-implant view, in contrast to Brandeis, an implant-sporting hacker.

Advertisement

In addition to being a bartender, Donovan is also an information broker, luring secrets out of clients through mixing signature cocktails. Brandeis and Donovan stumble upon a plan by megacorporation Supercontinent that involves making secret changes to people’s implants—or rather, the plan stumbles upon them, in the form of an AI who crashes through the bar’s door one night. From there, the game becomes a tangle of hacking, social engineering, and heavy drinking.

You spend most of your time in The Red Strings Club mixing cocktails. When a character comes into the bar, they’ll have several possible emotions—pride, depression, lust, euphoria—represented by icons in different positions on the screen. The player has to mix alcohol to move an indicator over the emotion they want to access. The labels on the bottles helpfully integrate arrows to remind you what moves where, and you later gain the ability to make the indicator move diagonally or to tilt its orientation. The controls are imprecise and uncomfortable, though it’s fun to spill booze everywhere. Nevertheless, it’s an unusual, enjoyable minigame, enriched with satisfying sound design. I regularly hurled ice cubes to the floor just to listen to them clatter.

The mixology screen. The bottle design is really great.

Donovan uses these cocktails to manipulate characters into telling him what he wants to know. You have a series of objectives to uncover before taking on Supercontinent. Questions like who their CEO really is, or what role the android you found plays, can be teased out of a prideful inventor or scared out of a depressed marketing executive if you read their starting state right and adjust it accordingly. Manipulating characters through alcohol might seem deceitful, but everyone who comes into the bar wants to get drunk to change how they feel. I struggled with the idea of forcing patrons to feel unpleasant emotions, but Donovan’s intentionality felt less like selfishness and more like acknowledging the truth behind why we drink.

Advertisement

You’ll need whatever information you can get for the game’s climax, an epic tangle of social engineering done over a landline phone while the evil corporation closes in. It feels like the perfect combination of what you’ve been doing all along: teasing, lying, considering and exploiting the connections between people. The Red Strings Club humanizes each of its villains and protagonists, and as I called up one person pretending to be their crush in order to trick them into giving me their computer password, I felt ashamed of how clever I thought I was being, and guilty about how easy it was to get what I needed.

Your choices are tracked by a screen that gets progressively more covered in red lines as the game goes. These “red strings” are another recurring theme: the web of secrets, lies, desires, and dreams that Donovan manipulates to get what he wants.

Advertisement

Things started fairly linear for me, though they soon became an intriguing mess. The Red Strings Club has a Telltale-style indicator for when a choice you’ve made has an impact, but what that impact would be seldom felt immediately apparent. Unlike Gods Will Be Watching, conversations never felt like they had a hard fail state. The game would tell me I’d done something impactful and a string would be added to my running tally, but it mostly felt like I was following my natural inclinations. With the exception of one moment, in which Brandeis has to approach another character in a methodical, clearly-laid out dance, The Red Strings Club’s choices are open-ended and vague, made in dialogue trees full of lines that run the gamut of options. It feels messy and mysterious in a very human way.

Choices have been blurred out to avoid spoilers.

The Red Strings Club deals with all the heavy issues you’d expect from cyberpunk: free will, humanity, technology, immortality. On occasion it felt a bit sophomoric—at one point two characters argue about how depression is vital to making good art—but it was just as quick to disagree with itself and come back down to earth. It’s a deeply emotional game, with a queer love story at its center. It’s also unexpectedly diverse, considering issues related to race, gender, and sexuality all as tacit parts of its future. One playthrough took me about three and a half hours, and I’m curious to see the consequences of different choices on another playthrough. A character will still probably fall out of a window, but it will mean something different depending how I get there.

", "url": "https://kotaku.com/a-cyberpunk-game-where-you-manipulate-people-with-alcoh-1822305786", "title": "A Cyberpunk Game Where You Manipulate People With Alcohol", "date_modified": "2018-01-22T21:02:00.000Z" }, { "content_html": "Comments", "url": "https://www.nytimes.com/2018/01/10/science/dolphins-self-recognition.html", "title": "Dolphins Show Self-Recognition Earlier Than Children", "date_modified": "2018-01-14T11:28:25.000Z" }, { "content_html": "Comments", "url": "https://news.ycombinator.com/item?id=16138969", "title": "Ask HN: As an adult introvertish nerd what makes you happy?", "date_modified": "2018-01-13T09:58:12.000Z" }, { "content_html": "Comments", "url": "https://www.nature.com/articles/s41598-017-15736-4", "title": "Two genes in Chromosomes 13 and 14, linked to Homosexuality", "date_modified": "2018-01-11T03:36:53.000Z" }, { "content_html": "
[Image: naagaoshi]

Quick question: During which season does Japan look most beautiful? Spring is probably the number one choice, followed by fall. But winter certainly is no slouch.

Twitter user Naagaoshi has been uploading a series of stunning Japanese winter photos, showing off just how lovely the country can look covered in snow and ice.

Have a look for yourself!

Kotaku East is your slice of Asian internet culture, bringing you the latest talking points from Japan, Korea, China and beyond. Tune in every morning from 4am to 8am.

", "url": "https://kotaku.com/the-beauty-of-japan-in-winter-1821903686", "title": "The Beauty of Japan In Winter", "date_modified": "2018-01-09T12:30:00.000Z" }, { "content_html": "Comments", "url": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4522609/", "title": "Psychological and psychiatric terms to avoid", "date_modified": "2018-01-08T01:40:29.000Z" }, { "content_html": "Comments", "url": "http://www.theage.com.au/federal-politics/political-news/advertising-banned-drinks-taxed-vending-machines-removed-doctors-plan-for-war-on-sugar-20180105-h0duw0.html", "title": "Doctors' plan for war on sugar", "date_modified": "2018-01-07T21:13:49.000Z" }, { "content_html": "\n\n\n\n www.racked.com/2017/12/27/16307098/bach…\n

\n Continue reading…\n

", "url": "https://www.racked.com/2017/12/27/16307098/bachelor-instagram-influencers-ads", "title": "The sneaky allure of the Bachelor Instagram influencer", "date_modified": "2017-12-27T16:19:17.000Z" }, { "content_html": "
\n\n\n\n\n\n
\n

\n Tlohtzin Espinosa and Owen Spence\n

\n \n\n

The original symphonic treatment for the game Destiny was long thought lost, thanks to it being shelved after a major staffing shake-up at developer Bungie. But Destiny fans received quite the Christmas miracle—albeit a legally dubious one—when fans discovered and posted an apparent rip of the album in question, titled Destiny: Music of the Spheres.

\n

The 8-track, 48-minute album leak, which is live as of press time at more than a few mirrors, was quickly confirmed as legitimate by two major contributors to the project: former Bungie composer Marty O'Donnell and former Bungie creative director Joe Staten. O'Donnell offered a \"I think this is it\" on Monday via Twitter, followed by an emphatic post of \"Finally! #NeverForgotMotS.\" Staten followed up with acknowledgement that Sir Paul McCartney himself sang a lyric Staten had suggested, then added, \"Glad #MOTS is finally out for all to hear.\"

\n

Read 5 remaining paragraphs | Comments

\n \n
", "url": "https://arstechnica.com/?p=1237433", "title": "Lost Destiny symphonic album, complete with Paul McCartney, has totally leaked", "date_modified": "2017-12-27T10:30:52.000Z" }, { "content_html": "

In late 2013, veteran Bungie composer Marty O’Donnell finished Music of the Spheres, an eight-part musical work designed to be released alongside Destiny. It never came out. But today, thanks to an anonymous leaker, the elusive work is finally on the internet—at least until the copyright strikes hit.

Composed by O’Donnell, his partner Michael Salvatori, and former Beatle Paul McCartney, Music of the Spheres was envisioned as a musical companion to Bungie’s ambitious Destiny. But Bungie and O’Donnell spent nearly a year battling over, among other things, publisher Activision’s failure to use O’Donnell’s music in a trailer at E3 2013. In April 2014, Bungie fired O’Donnell, and despite O’Donnell’s hopes, the company indefinitely shelved Music of the Spheres. He has made several public comments on the work since, and last month, he implicitly encouraged people to share it.

Advertisement

“Years ago, when I was Audio Director at Bungie, I gave away nearly 100 copies of Music of the Spheres,” O’Donnell tweeted on November 30, 2017. “I don’t have the authority to give you permission to share MotS. However, no one in the world can prevent me from giving you my blessing.”

In late 2016, teenager Owen Spence started his own independent project to recreate Music of the Spheres (as well-documented in this Eurogamer piece) using publicly available material. Spence has been in touch with me since then, and yesterday he told me via Twitter DM that he and a friend, Tlohtzin Espinosa, were contacted by someone with a copy of Music of the Spheres who wanted it to be public.

Advertisement

So they put it on Soundcloud, where you can now listen to it all (until Activision takes it down).

O’Donnell has not yet publicly commented on the leak—I’ve reached out for his thoughts—but it’s clear that he’s wanted this to happen for years now. Must be one hell of a Christmas gift.

UPDATE (7:50pm): In an e-mail, O’Donnell told me that he’s thrilled about this development:

I’m quite relieved and happy. This was the way it was supposed to have been heard 5 years ago.

My wife and I spent the afternoon with my now 93 year old father and we showed him that people were finally able to hear this work. It made our Christmas even better. My mother, his wife of over 60 years died a couple years ago and although she loved listening and shared it with some of her friends (she was a musician) she never understood why it wasn’t released.

I don’t know who actually did it but they have my blessing. I honestly don’t know how anyone could begrudge this any longer.

", "url": "https://kotaku.com/four-years-later-destinys-music-of-the-spheres-has-lea-1821572335", "title": "Four Years Later, Destiny's Music Of The Spheres Has Leaked [UPDATE]", "date_modified": "2017-12-25T19:00:00.000Z" }, { "content_html": "
Illustration by Chelsea Beck/GMG

Right after reinventing existing public services with private apps, hacking death may be the ultimate dream of SiliconValley’s elite. Death is truly the final boss for anyone who thinks enough money and lines of code can solve anything, and boy are they attacking it hard. In 2016, Mark Zuckerberg and his wife Priscilla Chan pledged $3 billion toward a plan to cure all diseases by the end of the century.

“By the time we get to the end of this century, it will be pretty normal for people to live past 100,” Zuckerberg said in 2016.

Advertisement

And to be sure, science, medicine and unlocking more about how the body functions have already worked what would look like a miracle to someone living centuries ago: the average life expectancy for someone born in the United States doubled in just 130 years, from 39 years in 1880 to 78 years in 2011. So Zuckerberg’s prediction may actually be easier than ridding his platform of Russian bots. Longevity—and potential immortality—is a particularly popular obsession with the tech world and Silicon Valley billionaires, who seem to be offended that death would ever get the better of them, and that somehow future generations MUST be able to bask in their immortal wisdom, even if their bodies are just throbbing electric impulses in a jar sustained by regular infusions of monkey testicles (yes, a real thing people tried for awhile).

The ultimate problem is that human bodies, these sad, slumping, failure-prone products of evolution, just aren’t cut out for living forever. People throughout history have tried, but the garbage body always gets in the way.

Advertisement

“We humans, as we are now, messy bags of blood and bone, are not really fit for immortality,” Stephen Cave, a philosopher at the University of Cambridge and author of the book Immortality: The Quest to Live Forever and How It Drives Civilization, told me. “So some really profound thing has to happen if we’re going to [change that].”

But if you’re interested in trying, oligarchs, rich lunatics and scientists throughout history provide some a framework, and a lot more is in the works at this very moment. Below, a rundown of the different approaches that have been taken up in the never-ending quest for life to never end.

Hack all diseases out of existence

Zuckerberg, along with his Silicon Valley pals from Google and 23andme, set up the Breakthrough Prize in 2012 to celebrate and promote science innovations, including fighting disease and living longer.

Advertisement

He also set up The Chan Zuckerberg Initiative, which will donate $3 billion over a decade to basic medical research with the goal of curing disease. Some have argued this approach isn’t the most efficient and the money would be better spent targeting single diseases at a time instead of an across-the-board assault. For instance, eradicating smallpox cost just $300 million in less than 10 years.

There is a problem with this approach, said Brian Kennedy, the director of the Center for Healthy Aging at the National University of Singapore: even if you treat diseases, you still haven’t cured aging itself.

“We don’t do healthcare [in the medical community], we do sick care,” he said, pointing out that the goal shouldn’t be just giving rich people access to cures for any disease but rather fundamentally attacking “aging” itself as a threat.

Advertisement

“Aging is the biggest risk factor to all these diseases that go out of control,” he said. “This is not just about a few billionaires living longer. This is about a million people living longer.”

Aging itself creates risks, he said, because organs and body systems inevitably break down over time. His center is researching ways to halt aging at the enzyme level. One of the most promising is the TOR pathway, a kind of cellular signaling that tells a cell to grow and divide or hunker down and turn up stress responses. Scientists believe that manipulating that pathway could slow down aging.

“It’s a really robust effect,” Kennedy said.

Once people realize that, he hopes his cause will be as flashy and imagination-capturing as Zuckerberg’s longevity quest.

Advertisement

“The most important thing we can do right now is to validate [the idea that] we can affect the aging,” he said. “Once that happens, I think interest level will go way up.”

Biohacking will also open up new avenues—and intense ethical debates—about what lengths people can go to to change their genetic code. Scientists, for instance, are still carefully exploring CRISPR (Clustered Regularly Interspaced Short Palindromic Repeats) technology, which acts like a homing missile that tracks down a specific DNA strand, then cuts and pastes a new strand in its place. It can be used to alter just about every aspect of DNA. In August, scientists for the first time in the United States used the gene editing technology on a human embryo to erase a heritable heart condition.

Harvesting young body parts

Throughout history, people have seized on the idea that you can essentially patch or infuse the human body with parts of other bodies and cheat death, kinda like jailbreaking your iPhone so it can accept any software.

Advertisement

Take, for instance Serge Voronoff, a Russian-born scientist who in the early 20th century believed animal sex glands held the secret to prolonging life. In 1920, he tried it out, taking a piece of monkey testicle and sewing it to a human’s (although, it should be noted, not his own) scrotum. The idea seemed to catch on: by the mid-1920s, according to Atlas Obscura, 300 people underwent his procedure; at least one woman received a graft of monkey ovary.

“The sex gland stimulates cerebral activity as well as muscular energy and amorous passion,” Voronoff wrote in his 1920 book, Life; a Study of the Means of Restoring Vital Energy and Prolonging Life. “It pours into the stream of the blood a species of vital fluid which restores the energy of all the cells, and spreads happiness.”

Advertisement

Voronoff eventually built his own monkey enclosure on his property and claimed he was able to restore 70 year olds to their youthful vigor. Some could live to 140, he claimed. He was able to charge as much as an average year’s salary at the time for the procedure.

Voronoff died in 1951, apparently never having rejuvenated himself.

Monkey testicles have fallen out of style, but, unlike the good doctor Voronoff, the idea of harvesting body parts is still very much alive.

Advertisement

Trump surrogate, Gawker killer and overall too-rich person Peter Thiel has talked about his interest in parabiosis, the process of getting transfusions of blood from a younger person, to reverse aging.

“I’m looking into parabiosis stuff, which I think is really interesting. This is where they did the young blood into older mice and they found that had a massive rejuvenating effect,” he told Inc. “It’s one of these very odd things where people had done these studies in the 1950s and then it got dropped altogether. I think there are a lot of these things that have been strangely under-explored.”

Studies have shown this may just be the latest snake oil tactic, though targeted to lunatic rich people who can’t help but be fascinated by the idea of literally feeding off the young.

Advertisement

It certainly didn’t work out for Alexander Bogdanov, a science fiction writer, doctor, and pioneer of cybernetics who dabbled in blood transfusions in the 1920s. He thought that if he ran a train of blood transfusions on himself, he could become functionally immortal. This thirst for blood met a hubristic end: he eventually took a blood transfusion from a malaria patient. The patient survived, but he did not.

Redefining the soul

Cave’s book breaks up immortality schemes throughout history into four classifications: the first one, staying alive in the body, involves all those life extending medicines and life hacking gene therapies discussed above. The second one involves resurrection, an idea that has fascinated people throughout history, from Luigi Galvani’s 18th century experiments running electricity through a dead frog’s legs to more recent efforts at cryonics, the process of freezing your body with the hope that future medicine or technology will be able to restore you to health. Some in Silicon Valley are interested in new versions of cryonics, but so far it doesn’t seem to be getting that much attention.

Advertisement

Cave’s third path involves finding immortality through the soul, something that has driven religious wars and controlled populations for eons. It takes as a fact that your physical body is a degrading mess that will one day betray you, but that doesn’t matter, since the soul is the real, eternal essence of who you are. But it’s best left to religious discussions nowadays, as science can’t seem to prove it exists.

“If bits of your brain are damaged, then bits of you, the fundamental deepest idea of who you are, have disappeared,” Cave said. He’s talking about the idea that if the soul is the indestructible essence of you that can survive eternity, why does our essence change when we suffer brain damage or other personality altering maladies? If your soul lets you live forever, which version of you exactly is the one that lives forever?

“That leads us to wonder if your soul is somehow supposed to be maintaining all these things, why can’t your soul do that for you? If it can do that when your whole brain is gone, why can’t that do that when a part of your brain is gone?”

Advertisement

But some techies argue the nature of these projects will redefine what a soul is entirely: not so much a ghostly essence of your being connected to a higher power, but more a specific set of brain signatures unique to you, a code that can be hacked like any other code.

“Consider, then, the modern soul as the unique neuronal-synaptic signature integrating brain and body through a complex electrochemical flow of neurotransmitters. Each person has one, and they are all different,” Marcelo Gleiser, a theoretical physicist, writer, and a professor of natural philosophy, physics and astronomy at Dartmouth College, wrote for NPR in April. “Can all this be reduced to information, such as to be replicated or uploaded into other-than-you substrates? That is, can we obtain sufficient information about this brain-body map so as to replicate it in other devices, be they machines or cloned biological replicas of your body?”

Advertisement

Google’s lifespan-extending project Calico launched in 2013 with a mission statement that calls aging “one of life’s greatest mysteries.” Also a great mystery is exactly what Calico has been up to: the company’s work has been shrouded in secrecy, which has led to lots of curiosity and frustration from the rest of people in the anti-aging field. So far, according to a New Yorker piece in April, all that’s known is the company is tracking a thousand mice from birth to death to find “biomarkers” of aging, what can be described as biochemical substances whose levels predict death. The company has invested in drugs that may help fight diabetes and Alzheimer’s.

Creating a lasting legacy

The tech side of things brings us to Cave’s fourth path to immortality: legacy. For ancient civilizations, that meant creating monuments, having your living relatives chant your name after you’re gone or carving names on tomb walls.

Advertisement

“If your name was spoken and your monuments still stood, they thought,” he wrote in his book, “then at least a part of you still lived.”

Today’s legacies look different than giant stone shrines, but the ego behind them is probably comparable. The idea of uploading consciousness to the cloud has crossed from science fiction into science possible: Russian web mogul Dmitry Itskov in 2011 launched the 2045 Initiative, an experiment to make himself immortal within the next 30 years by creating a robot that can store a human personality.

“Different scientists call it uploading or they call it mind transfer. I prefer to call it personality transfer,” Itskov told the BBC last year.

Fears of an immortal tech bro planet

So here is one of the obvious main problems with Silicon Valley-led innovations, like many other tech-based lurches into the advanced future: it could be too expensive for everyone to afford. Which in turn could mean that we’ll have a class of near immortals, or cloud-based consciousnesses, ruling over people bound to their horrifying analog bodies. The meshing of human/computer/nantech parts will also open up a whole new thinkpiece industry about when someone stops becoming a “person” all together and is just lines of code.

Advertisement

Kennedy said opening these options up to everyone will depend on what avenue of research proves the most effective. If aging is treated as a disease (and healthcare in general somehow becomes affordable to everyone), there’s hope.

“The challenge is to figure out ways to improve health span and get it to everybody as quickly as possible,” he said. “If it’s drugs, it’s achievable. If it’s a bunch of transfusions of young blood, that’s less achievable.”

If all this has you bristling at the thought of techies creating their own super race of “disruptors” impervious to the torments of time and the limits of flesh, that’s understandable. But Cave said you may be encouraged by the entire history of people who’ve chased extended lifespans, from ancient Egypt to the people clinging to their diets and exercise throughout the 21st century.

Advertisement

“The one thing that everyone has pursued immortality has in common,” he said, “is that they’re now six foot under, pushing up daisies.”

", "url": "https://lifehacker.com/how-to-outsmart-your-gross-body-and-live-forever-1820751558", "title": "How to Outsmart Your Gross Body and Live Forever", "date_modified": "2017-12-04T14:00:00.000Z" }, { "content_html": "Comments", "url": "http://fallibleideas.com/taking-children-seriously", "title": "Taking Children Seriously", "date_modified": "2017-12-03T08:50:33.000Z" }, { "content_html": "Comments", "url": "https://ventspace.wordpress.com/2017/10/20/games-look-bad-part-1-hdr-and-tone-mapping/", "title": "Games Look Bad: HDR and Tone Mapping", "date_modified": "2017-10-23T16:54:02.000Z" }, { "content_html": "
Photo by Christopher Burns

From an outsider’s perspective, what’s a cult and what’s not a cult can seem obvious. Not a cult: your new book group. Cult: that group your second cousin joined where all the women are renamed Meadow and are betrothed to Jeremy, their unshowered leader. Simple!

In reality, cults aren’t always so obvious; sometimes the cult-like aspects of an organization reveal themselves to you slowly, when you’re already fully invested.

Advertisement

In our latest podcast episode, Rick Alan Ross outlined the three criteria of establishing whether or not a group is a cult—as identified by psychiatrist Robert Jay Lifton.

1. There is an authoritarian figure in charge of the group who is revered like a god. Everyone and every decision revolves around said figure.

Advertisement

2. People in the group act against their own best interests, but in the best interest of the group (and the charismatic leader). This occurs through a process called “thought reform.”

3. The group exploits its members. The degree of harm inflicted on each member varies wildly depending on the group—some may take your money, others might inflict physical and sexual abuse.

Here’s the paper by Robert Jay Lifton in its entirety—it’s fascinating and worth a read, especially if there are any organizations in your life you have doubts about. (Your book group leader is strangely charismatic, and everyone always loves her book suggestions...)

", "url": "https://lifehacker.com/how-to-figure-out-if-youre-in-a-cult-1819678625", "title": "How to Figure Out if You're in a Cult ", "date_modified": "2017-10-19T18:30:00.000Z" }, { "content_html": "
Image via The Associated Press

In this episode we discussed cults: how they operate, how you identify one, what it’s like to be in one, and how to get out. To that end, we spoke with author Rebecca Stott, whose book In the Days of Rain: A Father, a Daughter, a Cult details her childhood in the Exclusive Brethren, a cult that believed the world is ruled by Satan. We also talked to Rick Alan Ross, the founder and Executive Director of The Cult Education Institute. And we talked with Elizabeth Yuko, a bioethicist and journalist who’s written extensively about cults.

Listen to The Upgrade above or find us in all the usual places where podcasts are served, including Apple Podcasts, Google Play, Spotify, iHeartRadio, Stitcher, and NPR One. Please subscribe, rate, and review!

Discussed in This Episode

We reached out to any organizations mentioned by Rick Ross if he either referred to them as a cult or described having received complaints about them. We received responses from two organizations. Jehovah’s Witnesses declined to comment directly, but sent us these links: Are Jehovah’s Witnesses a Cult? Are Jehovah’s Witnesses an American Sect?

We also heard from Landmark, who commented, in part:

Landmark is a global personal and professional growth, training, and development company that delivers programs and courses that empower and develop people to fulfill on what’s really important to them. Recognized for their leading-edge material and methodology, Landmark’s programs equip people to produce breakthrough results in areas such as career, relationships, productivity, and overall quality of life. More than 2.4 million people in more than 20 countries have taken Landmark’s programs and Landmark is considered one of the leading companies in its field.

Our Upgrades of the Week

Every week we like to let you in on the upgrades we’ve made in our own lives. This week we talked about bloomers, killing fruit flies, and this mesmerizing podcast.

Want to Say Hi?

There are two ways to reach out:

We look forward to hearing from you!

", "url": "https://lifehacker.com/how-to-identify-a-cult-with-rick-alan-ross-1819557884", "title": "How to Identify a Cult, With Rick Alan Ross ", "date_modified": "2017-10-17T12:30:00.000Z" }, { "content_html": "
[Image: Man-san’s Photo Gallery]

Every business has a starting point. For Japanese arcades, one of them was on department store rooftops.

Yes, Japanese arcades have their roots in the carnival type games often seen at local religious festivals. But way before coffee houses installed tabletop cabinets to capitalize on the late 70s Space Invaders craze, there were game machines atop department stores in cities like Osaka and Tokyo.

[Image: Man-san’s Photo Gallery]

Case in point: Namco, which still operates a large number of arcades in Japan. The company got its start making attractions, rides, and games for department store roofs. In 1955, when Namco was still Nakamura Manufacturing, the nascent company built two wooden horses for a Yokohama department store. This was not the first of its kind. The maiden rooftop amusement area opened on a Tokyo department store in 1903, with wooden horses, a seesaw and an indoor play area.

This image dates from 1931. Check out the rooftop amusement park on top of this Tokyo department store. 「今日の東京スカイツリーと松屋浅草、あづち… via Namazu-tron | CC]

By the time Nakamura Manufacturing entered the scene in the mid-1950s, this was a well established tradition in Japan, with a history of buildings covered with ropeways, Ferris Wheels and other attractions. No wonder they’re known as okujou yuuenchi (屋上遊園地), literally “rooftop amusement park.”

[Image: Tetsugaku News]

But with cities rebuilding after World War II, these rooftop play areas offered new business opportunities.

[Image: Tetsugaku News]

In the early 1960s, Nakamura Manufacturing constructed a kiddy attraction called “Roadway Ride” on top of Mitsukoshi Department Store with children “driving” small cars on a railroad type track.

[Image: Namco]

Not everything was a ride, as there were also coin operated and carnival type games. Other companies, like Sega, also made mechanical games that were enjoyed on these rooftops.

Advertisement

Website Tetsugaku News recently published a series of old photos of Japanese department store roofs, showing the different attractions.

[Image: Tetsugaku News]
What a massive amusement area. [Image: Tetsugaku News]
[Image: Tetsugaku News]

Via website Nippon Sumizumi Kanko, here are some of the retro games found on the rooftop play area of Nagasaki’s Hamaya Department Store.

[Image: Nippon Sumizumi Kanko]
[Image: Nippon Sumizumi Kanko]
[Image: Nippon Sumizumi Kanko]

Around this same time, there were also bowling alleys with mechanical games, but everything radically changed with Space Invaders. Dedicated establishments, called “inbeedaa hausu” (“invader house”), starting popping up across the country, evolving into the Japanese gaming arcades of today.

Advertisement

When I came to Japan in 2001, rooftop arcades were still fairly common. But these days, with more and more of them are being shuttered instead of getting needed updates and repairs.

For example, the previously mentioned Hamaya Department Store’s rooftop amusement park is no more, and after 56 years in operation, the okujou yuuenchi on Hanshin Department Store in Osaka was shut down, as documented by Man-san’s Photo Gallery. These are just two of many, and it’s increasingly rare for department stores to have them. Sadly, the age of rooftop amusement parks is drawing to a close.

[Image: Man-san’s Photo Gallery]
[Image: Man-san’s Photo Gallery]

But many modern Japanese arcades still have this rooftop amusement park DNA, offering small rides for children. Some, like Joypolis, are modern throwbacks to the okujou yuuenchi of yore.

The only thing that is missing is the rooftop setting.

For more photos, check out Nippon Sumizumi Kanko and Man-san’s Photo Gallery.

This article was originally posted on April 6, 2017.

Kotaku East is your slice of Asian internet culture, bringing you the latest talking points from Japan, Korea, China and beyond. Tune in every morning from 4am to 8am.

", "url": "https://kotaku.com/an-unlikely-origin-of-japanese-arcades-1794067285", "title": "The Birth Of Japanese Arcades ", "date_modified": "2017-10-10T11:00:00.000Z" }, { "content_html": "Comments", "url": "http://www.spoon-tamago.com/2017/10/04/japanese-vending-machines-at-night-juxtaposed-with-a-wintry-hokkaido-landscape/", "title": "Japanese Vending Machines at Night Juxtaposed with a Wintry Hokkaido Landscape", "date_modified": "2017-10-05T19:41:27.000Z" }, { "content_html": "
GIF
[Images: Tofugu]

A big part of Your Name takes place in Tokyo. Over on Tofugu.com, Kanae Nakamine visited the real-world locations depicted in the movie. 

The resulting comparisons between the anime and real life are fascinating.

Nakamine also created a helpful itinerary in case you want to make a Your Name seichijunrei (聖地巡礼) or pilgrimage, visiting the places depicted in the hit anime.

[Images: Tofugu]
[Images: Tofugu]
[Images: Tofugu]
[Images: Tofugu]
[Images: Tofugu]
[Images: Tofugu]
[Images: Tofugu]

Be sure to check Tofugu for more comparisons and info regarding where these Tokyo spots are located.

In case you missed it, you can read Kotaku’s Your Name review right here. 

Kotaku East is your slice of Asian internet culture, bringing you the latest talking points from Japan, Korea, China and beyond. Tune in every morning from 4am to 8am.

", "url": "https://kotaku.com/your-names-anime-locations-compared-with-the-real-world-1819130993", "title": "Your Name's Anime Locations Compared With The Real-World Ones", "date_modified": "2017-10-04T11:00:00.000Z" }, { "content_html": "

Here’s a cool video tribute to Half-Life 2: Episode 3 by Anomalous Materials. Oh, what could have been.

", "url": "http://steamed.kotaku.com/1801256933", "title": "Here’s a cool video tribute to Half-Life 2: Episode 3 by Anomalous Materials.", "date_modified": "2017-09-07T01:10:40.000Z" }, { "content_html": "

There’s an official Evangelion x New Balance line of sneakers coming out in Japan and China this weekend.

There’ll be four colours available, each in a very subtle scheme, while they’ll also come in nice padded zip-up boxes that feature Evangelion imagery.

They’ll go on sale for around USD$100. More pics at Hypebeast.

", "url": "http://kotaku.com/evangelion-sneakers-are-surprisingly-tasteful-1800674421", "title": "Evangelion Sneakers Are Surprisingly Tasteful", "date_modified": "2017-09-07T00:00:00.000Z" }, { "content_html": "Comments", "url": "http://www.telegraph.co.uk/books/non-fiction/spot-psychopath/", "title": "How to spot a psychopath", "date_modified": "2017-09-02T15:59:57.000Z" }, { "content_html": "Comments", "url": "https://news.harvard.edu/gazette/story/2017/08/when-it-comes-to-internet-privacy-be-very-afraid-analyst-suggests/", "title": "When it comes to internet privacy, be afraid, analyst suggests", "date_modified": "2017-08-29T15:23:57.000Z" }, { "content_html": "Comments", "url": "https://www.activism.net/cypherpunk/manifesto.html", "title": "A Cypherpunk's Manifesto", "date_modified": "2017-08-25T23:32:43.000Z" }, { "content_html": "Comments", "url": "http://www.marclaidlaw.com/epistle-3/", "title": "Half life 3 story released by Marc Laidlaw", "date_modified": "2017-08-25T08:35:35.000Z" }, { "content_html": "
My poor black Leviathan, so unhip.

I’ve been reviewing Razer accessories and hardware for ages, and aside from the odd license tie-in and those rainbow headsets, they’ve mostly had one thing in common—they’ve been black. Well now we’ve got the Mercury line, so bright and white they look like the restless spirits of real Razer products.

Razer’s actually got two new color schemes going on, Mercury (white) and Gunmetal (gray), but gray is really just light black, babysteps compared to the jarring contrast between Razer’s jet black lineup and these ivory beauties.

First off we have the Invicta “gaming surface.” I put gaming surface in quotes because it’s generally just a fancy term for mouse pad, but in this case it might be appropriate. For one, this has a case.

The Invicta is a dual-sided gaming surface (it’s growing on me). One side is smooth for speed. The other is rough, for control. It costs $60 and weighs around 1.5 pounds, not counting the case.

The weight is because the Invicta pad comes with an aluminum base plate.

That’s one serious mouse pad. And here’s a serious mouse for it. The Lancehead Tournament Edition is a mouse with a 16,000 DPI 5G optical sensor, tracking at 450 inches per second. As a man who generally gets by with a trackball, that’s ridiculous.

The $80 Lancehead is completely symmetrical, with side buttons on the left and right so anyone can use it, regardless of dominant hand. Plug it in and place it on top of the Invicta and it looks like a racing pod from some utopian future.

The Kraken 7.1 v2 is Razer’s middle-of-the-road headset, priced at $99.99 (the $200 Razer Tiamat 7.1 V2 is their latest top-of-the-line wired set). It’s a great set of cans with a pretty good retractable mic. The switch to white isn’t as dramatic for this one, mainly because I never see it once it’s on my head. It’s still very pretty.

The $150 Blackwidow X, on the other hand, barely looks like a Razer product anymore. Rather than go straight-up white everything, Razer made the exposed metal plate silver, giving the board a lovely contrast. The Razer logo on the front is barely visible when the unit’s LEDs are off.

There is something otherworldly about a set of shine-through white keycaps with soft RGB lighting piping through. It’s dreamy.

Razer’s even gone as far as running a batch of their custom switches (this one has the clicky greens) with a pale gray housing as opposed to the normal black. The company’s made a real commitment here.

The whole set comes together nicely. Razer is known for creating aggressive-looking gaming hardware with bold (some would say obnoxious) branding. The Mercury line softens those rough edges considerably.

Seriously looks like someone’s Razer desktop died and is haunting me.
", "url": "http://kotaku.com/razer-gear-looks-really-nice-in-white-1797933112", "title": "Razer Gear Looks Really Nice In White", "date_modified": "2017-08-17T14:00:00.000Z" }, { "content_html": "Comments", "url": "https://www.l4sb.com/lpa/anonymous-llc/", "title": "Anonymous LLC", "date_modified": "2017-08-05T01:03:25.000Z" }, { "content_html": "

Nier: Automata audio engineer Masami Ueda has written a cool blog post detailing how he implemented composer Keiichi Okabe’s secondary hacking soundtrack. It’s easy to overlook how much work something like that takes, and fun to get a look at Ueda’s process.

", "url": "http://kotaku.com/nier-automata-audio-engineer-masami-ueda-has-written-a-1797468934", "title": "Nier: Automata audio engineer Masami Ueda has written a cool blog post detailing how he implemented", "date_modified": "2017-08-02T16:45:00.000Z" }, { "content_html": "Comments", "url": "https://arstechnica.com/science/2017/07/defense-of-gwyneth-paltrows-goop-offers-case-study-on-how-to-sell-snake-oil/", "title": "Defense of Gwyneth Paltrow’s Goop offers case study on how to sell snake oil", "date_modified": "2017-07-17T03:34:19.000Z" }, { "content_html": "Comments", "url": "https://www.google.com/culturalinstitute/beta/exhibit/ogKCPmGdPtB7Iw", "title": "Tokyo street fashion and culture: 1980 – 2017", "date_modified": "2017-07-15T19:06:19.000Z" }, { "content_html": "
GIF

Released this week on Steam by Sand Sailor Studio, Black The Fall is a 2D side-scrolling puzzle platformer that shares many similarities with last year’s indie hit, Inside. Dark and moody atmosphere? Clever puzzles requiring plenty of trial and error? It’s got all that, plus a lil’ robot friend.

Comparisons between Black The Fall and Playdead’s dark and thoughtful platformer are unavoidable. The game takes place in a ruined world ravaged by communism. The player is a tired worker trying to escape years of toil under the regime. Though eventually opening up to the brighter outside world, the opening moments of the game take place within the gray walls and black shadows of a vast factory. Early in the game the player gains access to a laser pointer, a tool that manipulates mechanical devices as well as the brains of mind-controlled workers.

It’s all very dark and hopeless, until the player makes a new friend.

The introduction of this little robot pupper makes a dark game a bit brighter. The robodog can help the player climb to higher heights, activate electronic switches and can even act as a brace, as seen in the GIF atop this post.

Advertisement

I streamed an hour and a half of Black The Fall earlier this week. Check out the stream archive below to see my escape attempt panned out.

", "url": "http://kotaku.com/if-you-liked-inside-youll-like-black-the-fall-and-its-1796883369", "title": "If You Liked Inside, You'll Like Black The Fall And Its Little Robot Dog Too", "date_modified": "2017-07-13T16:00:00.000Z" }, { "content_html": "Comments", "url": "https://www.koipun.com/blog/how-i-learned-japanese-bryan-from-kuro-pixel", "title": "How I learned Japanese: Interview about habits and learning Japanese in Japan", "date_modified": "2017-06-27T16:04:44.000Z" }, { "content_html": "Comments", "url": "https://theoutline.com/post/1617/chiropractors-are-bullshit", "title": "Chiropractors are bullshit", "date_modified": "2017-06-25T12:49:07.000Z" }, { "content_html": "Comments", "url": "https://backlothelp.netflix.com/hc/en-us/articles/217237077-Production-and-Post-Production-Requirements-v2-1", "title": "Netflix Originals: Production and Post-Production Requirements v2.1", "date_modified": "2017-06-22T14:39:13.000Z" }, { "content_html": "Comments", "url": "http://bombmagazine.org/article/665269/toshio-masumoto-s-em-funeral-parade-of-roses-em", "title": "A restored masterpiece unmasks Tokyo's underground gay subculture of the 1960s", "date_modified": "2017-06-17T02:44:58.000Z" } ] }