Tim Callaghan

Two of AWS’ Sr. Principal Engineers, Nicholas Matsakis and Marc Bowes, take us inside the development of Aurora DSQL in a fascinating new blog. The post shows how the team scaled write operations without two-phase commit, overcame garbage collection hurdles, and ultimately embraced Rust for both data and control planes. What I find particularly interesting is how the team's journey with Rust evolved. They started cautiously, using it for a single component, and ended up rewriting almost everything in Rust. It's a reminder that sometimes the hard choices - like adopting a new language - can unlock tremendous value. The full post is well worth your time. It's not just about database internals, but also about team dynamics, a culture of learning (and questioning established approaches), and making difficult architectural decisions. Read it here: https://lnkd.in/eMhxBu9e Now, go build!

1,045

16 Comments

People often ask me about the differences in architecture between Amazon Dynamo (the 2007 SOSP paper), DynamoDB (the AWS serverless NoSQL database), and Aurora DSQL (the AWS serverless SQL databases). I captured the key points on durability, high availability, programming model, and trade-offs in my blog: https://lnkd.in/gkMG_tVD TL;DR: Despite the shared name, Dynamo and DynamoDB have little in common architecturally, and make very different trade-off decisions.

632

8 Comments

🎯 Running inference workloads on Kubernetes? I just wrote a deep dive on Gateway API Inference Extension and how it works. The Gateway API Inference Extension introduces two new Custom Resource Definitions (CRDs): InferenceModel and InferencePool. By using these two new resources, along with the endpoint selector, your L7 routing infrastructure becomes an “Inference Gateway” enabling you to self-host GenAI/LLMs with a “model-as-a-service” mindset. https://lnkd.in/g_eFqwUd #ai #llm #genai #inference

57

2 Comments

Things just got even more interesting in open source (database) land! Today, Redis makes a dramatic return to its open source roots, offering Redis 8 under the AGPLv3 license. The shift follows a similar move by Elastic in August 2024 and completes the company’s somewhat circuitous licensing path. In both cases, I suspect that neither company ever really wanted to leave open source. The irony is that the very thing that most threatened to hurt their businesses actually helped them: forking. Redis’ license change “achieved our goal,” writes Redis CEO Rowan Trollope. “AWS and Google now maintain their own forks.” How is this a good thing for Redis, developers, enterprise customers, and even Valkey? Read on. 😇 https://lnkd.in/gTJNnVeZ

27

4 Comments

Business processes are becoming complex enough that you can't run your company on just SAP, Oracle, and MuleSoft. Companies are investing heavily in AI workflows, which must integrate with legacy systems, existing microservices, and external API's. Open-source stacks are increasingly replacing the monolithic, proprietary stacks that were popular 5-10 years ago (especially before Kubernetes). These trends are only going to accelerate—and it means that you need to have a software engineering organization and architecture that is: - open-source - cloud-capable and cloud-agnostic - built for the demands of the modern enterprise. Colt McNealy and I have been working on such a platform, based on the open-source LittleHorse Kernel (and the surrounding Saddle ecosystem) which I am very excited about. You can check out the core Kernel on GitHub, and get in touch with Colt on Slack if you want to learn more! https://lnkd.in/gsEDVgDR

83

5 Comments

New blog post: A Fork in the Road: Deciding Kafka's Diskless Future. Kafka is getting serious about S3 and finds itself at an architectural crossroads that will shape its next decade. Several new KIPs (1150, 1176, 1183) aim to reduce replication costs across cloud availability zones, but the implications go far beyond networking cost. It’s a mistake to think of S3 as simply a cheaper disk or a networking cheat. Building on object storage opens the door to operational benefits such as elastic, stateless compute — something many modern analytics systems already exploit. In my latest post, I outline two competing future paths: 🔹 Evolutionary path: Reuse large parts of existing Kafka components to reduce code changes and long-term maintenance. 🔹 Revolutionary path: Separate stateless and stateful layers to realize the full operational benefits of disaggregated storage. The post examines the trade-offs of each path, how the current KIPs map to these two paths and poses a broader question: what should Kafka become? And what will keep it relevant in the decade ahead? https://lnkd.in/d4E2BAe4

149

12 Comments

🗣️ "By leveraging Debezium and Kafka Connect, we were able to stream real-time database changes efficiently without writing a single line of application code" Always loving to read a good report of folks using Debezium in production. Nice one, by Daniil Roman 👍. 👉 https://lnkd.in/eHemsnCF

130

4 Comments

Today, we're excited to announce Ubicloud PostgreSQL v2: Seven new features, higher performance! https://lnkd.in/eTDxw4bc How is this different than other managed Postgres solutions? 1. We use local NVMes. This helps us deliver price-performance that's 7.9 - 11.1x better than Aurora Postgres. 2. Our team has years of experience with Postgres. This makes us keep reliability as a forethought. 3. Ubicloud provides the first open-source solution for managed Postgres. This keeps you in control of your database. If interested, check out our v2 and tell us what you think! ⬇️⬇️⬇️

77

2 Comments

Exciting news for Amazon MSK users! AWS Lambda now natively supports Avro and Protobuf formatted events from Apache Kafka, including Amazon MSK, Confluent Cloud, and self-managed clusters—all in Provisioned Mode. This advancement marks a significant leap forward for event-driven architectures, as it eliminates the need for custom deserialization code and manual schema registry management. By integrating directly with AWS Glue Schema Registry, Confluent Schema Registry, or self-managed registries, Lambda can now validate and process Avro or Protobuf events seamlessly, allowing teams to focus more on delivering business value and less on infrastructure plumbing. The benefits extend well beyond simplified integration. Avro and Protobuf are renowned for their compact message sizes and rapid (de)serialization, offering notable performance improvements over JSON. With native support, Lambda users can choose to receive events in their original binary format for direct object mapping or as clean, validated JSON for easy integration across various programming languages. Tools like Powertools for AWS Lambda further streamline this process, making it effortless to work with business objects. Event filtering at the source is another standout feature. Teams can now set up rules to filter out irrelevant events before they reach Lambda functions, reducing costs and improving pipeline efficiency. When schemas evolve, Lambda automatically retrieves and applies the latest versions, ensuring functions always process up-to-date, validated data—with no code changes required. Robust error handling is also enhanced, as problematic events can be routed to #SQS, #SNS, or #S3 for debugging and analysis, keeping workflows reliable and resilient. #AWS #lambda #MSK #ApacheKafka #avro #protobuf https://lnkd.in/gutjCjkD

62

3 Comments