Greg Yang

In-memory databases are the architectural backbone of real-time AI. They’re a necessary advantage for memory-intensive workloads that rely on instant access to context. Modern systems like Redis blend durability, multi-model flexibility (JSON, vectors, time-series), and tiered storage to deliver the speed of memory with the reliability of disk. The result? Real-time analytics, instant personalization, and AI systems that actually feel intelligent. Here’s how we deliver the speed of memory with the reliability of disk: https://lnkd.in/gSm65kdB

49

Anyscale 🤝GKE Through our continued collaboration with the Google Kubernetes Engine (GKE) team, we’re advancing the experience to scale Ray and Kubernetes in the cloud. This work includes: Label Selectors for improved scheduling: https://lnkd.in/gUZ8iVUW New Features for AI Scaling: https://lnkd.in/gAGSMiir Enhanced TPU Support: https://lnkd.in/gsKDvwpv

65

3 Comments

“1 million rows per second.” In his opening keynote, our CEO Max Liu walked through what that really means in production: - TiDB processing 1M rows/sec - 10x higher throughput for cross-region replication - A data layer built to make multi-tenant architectures actually practical at scale If you’re thinking about SaaS, multi-region, or AI-heavy workloads, this is the kind of infrastructure ceiling you want. 🎥 Watch the full keynote from Max Liu at #TiDBSCaiLE to see what’s behind the numbers: https://ow.ly/BgKl50YetBq #TiDB #DistributedSQL #MultiTenancy #CloudNative #Scalability

24

When Steve Huynh was a principal engineer at Amazon, meetings began with a "study hall." Amazon had a "reading culture" even among engineers, Huynh recently told the Pragmatic Engineer podcast, speaking of his time at the tech giant. Employees frequently drafted six-page memos, which they shared with the company to update progress and demonstrate new projects. Huynh, who said the company's embrace of writing and reading the memos was part of its "secret sauce," said employees' writing was often constrained to the format, whether it was a business strategy or press release. Huynh started at Amazon in 2006, only a few years after the company turned its first profit and while Jeff Bezos was at the helm. Read more about how Bezos instilled this culture of memo-writing from the top down on Business Insider: https://lnkd.in/gBqbEf_j (Credit: Daniel Berman) #amazon #reading #jeffbezos

216

2 Comments

From the start, Metronome knew real-time billing couldn’t be built on batch. That’s why they chose Confluent as the foundation for their data architecture, streaming billions of events daily and processing tens of thousands of invoices per second with ease. We handle the infrastructure complexity so 90% of their engineers can focus on building and improving the product. Explore how they’re delivering a modern billing experience powered by real-time data infrastructure → https://cnfl.io/40S0YDS

75

The most effective architectural changes are often subtractive - removing entire layers of cost and complexity. Behind the scenes, OpsHelm, Inc.'s streaming backbone was migrated from a costly MSK and unreliable NATS setup to a single diskless Kafka architecture. The results were immediate: - Annual spend dropped from >$50,000 to <$10,000. - Data loss incidents went from recurring to zero. - Manual scaling and cross-cloud networking fees were eliminated. Read the full story of their migration and 78% cost reduction: https://lnkd.in/dS6UK9-7 #disklesskafka #datastreaming #Aiven

31

Sealos runs 6,000+ MySQL, Redis, MongoDB, and Kafka clusters across 4 regions using KubeBlocks operators on Kubernetes. Learn how CRDs, automated HA, backup, drift detection, and PITR make one engineer manage DBaaS at scale. More: https://ku.bz/v1BQHKg1k

29

1 Comment

“In a very binary sense, Metronome made self-service at Anyscale possible.” That’s Matt Weber, Software Engineer, reflecting on Anyscale's expansion into new revenue channels. As the team added PLG and cloud marketplaces (AWS, GCP, and Azure), their manual billing setup couldn’t support the volume or complexity. Reporting, reconciliation, and manual workflows were slowing them down. With Metronome as their billing foundation, Anyscale unified every revenue motion on one platform and accelerated growth across the business. They launched self-serve billing end-to-end, brought AWS, GCP, and Azure marketplaces live in under three weeks each, and freed up three engineers and a PM from dedicated billing work. 🎉🤝 Big congratulations to the Anyscale team on the expansion—this is what it looks like when innovative product and modern monetization infrastructure come together. Read the full story at the link in the comments.

37

1 Comment

Scaling systems is fun. Until something breaks quietly 🥲. Our Kafka brokers started showing TCP listen drops across regions. No crashes. No CPU spikes. No obvious red flags. Just silent degradation in connection reliability 🫠. At Freshworks, we run self-managed Apache Kafka clusters across 5 AWS regions—powering real-time data for thousands of customers. When reliability took a hit, we had to dig deep. Our senior staff engineer, Dinesh Kumar Murugiah explains how we tackled it: 🕵️ Investigated Linux TCP internals to pinpoint where connections were getting dropped 🛠️ Used eBPF tools to trace packet behavior without disrupting production 📊 Systematically ruled out network latency, SYN floods, sluggish clients, and broker delays 📈 Identified the real issue: a sharp spike in connection volume + a hardcoded Kafka queue limit The cause wasn’t malicious traffic—it was legitimate load outgrowing configuration defaults. Engineering at scale isn’t about firefighting. It’s about catching invisible failures before they affect users. Read the full story from the comments below 👇

105

2 Comments

Kikoff is constantly innovating to build smarter, faster financial tools. In an interview with Built In SF, Engineering Manager Patrick Wang shares how the team is tackling technical debt and optimizing performance to solve problems for customers, faster. Read more on what’s ahead: https://lnkd.in/gBvk7cQ9

10

Our friends over at Discord incredible engineering blog on how they're processing petabytes of data with dbt Labs and Dagster while supporting 100+ developers working across 2,500+ models! Key Highlights: 📦 Environment separation strategy to prevent developers from overwriting each other's test tables 📈 Performance optimizations that achieved a 5x speedup in dbt execution times 🧠 Smart versioning system for automated, efficient backfills ⚙️ Comprehensive macros and CI/CD guardrails for data quality Great work Chris D. ! Link in the comments.

51

1 Comment

From Terraform to Crossplane Kubernetes-native IaC is evolving—are you keeping up? Crossplane is redefining how we manage infrastructure: declarative, secure, and fully integrated with the K8s ecosystem. Think Terraform, but closer to your cloud-native apps. -YAML over DSL -Native RBAC & multi-tenancy -Seamless integration with Helm, Operators & more Explore how Crossplane offers the next-gen control plane—bridging DevOps and Platform Engineering like never before. #InfrastructureAsCode #Crossplane #Kubernetes #CloudNative #DevOps #PlatformEngineering #MSysTechByte #Infraunlocked #MSysTechnologies

13

We’re excited to announce that Anyscale's fully managed Ray platform can now be deployed directly within your CoreWeave account using Bring‑Your‑Own‑Cloud (BYOC) on CoreWeave Kubernetes Service (CKS). Powered by CoreWeave and Anyscale, this integration delivers: ✔️ Efficient MLOps pipelines: Launch managed Ray clusters faster—without the heavy DevOps lift. ✔️ Holistic observability: Monitor GPU health, job performance, and networking in real time. ✔️ Outstanding resource utilization: Capture up to 96% goodput for maximum compute efficiency. ✔️ Enterprise-grade governance: Features like SAML SSO, access control, and Weights & Biases integration deliver compliance with ease. Read more in our latest blog post: https://hubs.la/Q03CsvLY0 #AIinfrastructure #Ray #Anyscale #CoreWeave #MLOps #Kubernetes #AICloud

188

1 Comment

Traditional Kafka consumers for analytics = expensive, complex, and fragile. Replay, deduplication, schema drift… it’s a lot. On Oct 9, we’ll show how Estuary + MotherDuck simplify all of this with a fresh approach to streaming ingestion. 🔗 https://lnkd.in/dncWFDGm

8

You need tools that simplify, not complicate, your infrastructure setup. The #OpenSource Cluster Toolkit offers pre-built blueprints and modules for rapid, repeatable cluster deployments. You get easy integration with JAX, PyTorch, and Keras; platform teams get simplified management with Slurm, GKE, and Google Batch, plus flexible consumption models like Dynamic Workload Scheduler and a wide range of hardware options. Learn more about simplifying your cluster setup and deploying quickly using our Cluster Toolkit → https://goo.gle/4jlhEur

165

9 Comments

The engineering team optimized Kestra’s core to reduce CPU and memory usage, speed up executions, and cut down latency significantly. Benchmarks included: some workflows now execute nearly 90% faster. Details and benchmarks in our latest article: https://lnkd.in/eapvUZV3

33

1 Comment