Ricardo Ferreira

In-memory databases are the architectural backbone of real-time AI. They’re a necessary advantage for memory-intensive workloads that rely on instant access to context. Modern systems like Redis blend durability, multi-model flexibility (JSON, vectors, time-series), and tiered storage to deliver the speed of memory with the reliability of disk. The result? Real-time analytics, instant personalization, and AI systems that actually feel intelligent. Here’s how we deliver the speed of memory with the reliability of disk: https://lnkd.in/gSm65kdB

49

🎯 Running inference workloads on Kubernetes? I just wrote a deep dive on Gateway API Inference Extension and how it works. The Gateway API Inference Extension introduces two new Custom Resource Definitions (CRDs): InferenceModel and InferencePool. By using these two new resources, along with the endpoint selector, your L7 routing infrastructure becomes an “Inference Gateway” enabling you to self-host GenAI/LLMs with a “model-as-a-service” mindset. https://lnkd.in/g_eFqwUd #ai #llm #genai #inference

57

2 Comments

Hyland's Jitesh S. Ghai shares more about Enterprise Context Engine and Enterprise Agent Mesh. Hyland takes the hundreds of millions of artifacts across your fragmented application landscape and connects it with the industry's first Enterprise Context Engine and Enterprise Agent Mesh. Learn more here: https://bit.ly/3VoZTjD

197

1 Comment

From the start, Metronome knew real-time billing couldn’t be built on batch. That’s why they chose Confluent as the foundation for their data architecture, streaming billions of events daily and processing tens of thousands of invoices per second with ease. We handle the infrastructure complexity so 90% of their engineers can focus on building and improving the product. Explore how they’re delivering a modern billing experience powered by real-time data infrastructure → https://cnfl.io/40S0YDS

75

The most effective architectural changes are often subtractive - removing entire layers of cost and complexity. Behind the scenes, OpsHelm, Inc.'s streaming backbone was migrated from a costly MSK and unreliable NATS setup to a single diskless Kafka architecture. The results were immediate: - Annual spend dropped from >$50,000 to <$10,000. - Data loss incidents went from recurring to zero. - Manual scaling and cross-cloud networking fees were eliminated. Read the full story of their migration and 78% cost reduction: https://lnkd.in/dS6UK9-7 #disklesskafka #datastreaming #Aiven

31

We're donating Declarative Pipelines - a proven standard for building reliable, scalable data pipelines - to Apache Spark™. Based on our core declarative ETL framework, Spark Declarative Pipelines helps engineers tackle common challenges like complex pipeline authoring, operational overhead, and siloed batch and streaming workloads. https://lnkd.in/g7vn4ftp

584

10 Comments

New blog post: A Fork in the Road: Deciding Kafka's Diskless Future. Kafka is getting serious about S3 and finds itself at an architectural crossroads that will shape its next decade. Several new KIPs (1150, 1176, 1183) aim to reduce replication costs across cloud availability zones, but the implications go far beyond networking cost. It’s a mistake to think of S3 as simply a cheaper disk or a networking cheat. Building on object storage opens the door to operational benefits such as elastic, stateless compute — something many modern analytics systems already exploit. In my latest post, I outline two competing future paths: 🔹 Evolutionary path: Reuse large parts of existing Kafka components to reduce code changes and long-term maintenance. 🔹 Revolutionary path: Separate stateless and stateful layers to realize the full operational benefits of disaggregated storage. The post examines the trade-offs of each path, how the current KIPs map to these two paths and poses a broader question: what should Kafka become? And what will keep it relevant in the decade ahead? https://lnkd.in/d4E2BAe4

149

12 Comments

🗣️ "By leveraging Debezium and Kafka Connect, we were able to stream real-time database changes efficiently without writing a single line of application code" Always loving to read a good report of folks using Debezium in production. Nice one, by Daniil Roman 👍. 👉 https://lnkd.in/eHemsnCF

130

4 Comments

Cassandra and ScyllaDB rely on fundamentally different scaling models. But even in tests that favor Cassandra's vNodes model, ScyllaDB sustained 4X the throughput while maintaining consistently lower P99 latencies under similar infrastructure. See the results for yourself here > https://ow.ly/3SHB50Y1fL9 #ScyllaDB

25

Traditional Kafka consumers for analytics = expensive, complex, and fragile. Replay, deduplication, schema drift… it’s a lot. On Oct 9, we’ll show how Estuary + MotherDuck simplify all of this with a fresh approach to streaming ingestion. 🔗 https://lnkd.in/dncWFDGm

8

🔭 Why does observability belong in your DataOps platform? Kenten Danas, Senior Manager of Developer Relations at Astronomer, explores the challenges of using multiple tools for data pipeline observability, outlines what Apache Airflow offers for observability features (and notably, what it doesn't), and how Astro Observe delivers a comprehensive, single pane of glass for Airflow pipelines. Read the blog: https://bit.ly/3KtIbcI

28

For years, PostgreSQL has been one of the most critical systems powering products like ChatGPT and the API. In our latest Engineering Blog post, Bohan Zhang takes us under-the-hood on the novel ways we scaled PostreSQL to support an unprecedented 10x growth in production load in just one year. This work has been critical to ensuring both consumer and enterprise users have a fast, reliable experience. Learn more in at the link in the comments below.

324

17 Comments

We’re excited to introduce TransformWithState, a powerful new arbitrary state API for Apache Spark Structured Streaming. Now in Public Preview with DBR 16.2. With 90+ PRs over 10 months, this is the largest open-source contribution from the Streaming Engine team in three years. TransformWithState unlocks net new use cases for streaming like real-time SCD-1/SCD-2 processing and simplifies previously complex ones like custom session tracking, reducing the coding required by hundreds of lines. The new arbitrary stateful operator adds support for the following features: - Support for object oriented state processor for defining complex, custom logic with ability to define/remove state variables as needed - Support for composite types (such as list, map etc) with operations optimized for the underlying state storage - Support for timers and TTL  - Support for initializing state through an existing query checkpoint, Delta tables or any other source - Support for schema validation and schema evolution for enhanced flexibility around state schema - Support for reading state written with new operator using state data source reader API for easier debugging With this Apache Spark Structured Streaming leapfrogs other streaming systems in several areas but most importantly simplicity; customers can now build custom stateful operators using a familiar DataFrame API and without much setup. https://lnkd.in/gXaF6iRD

227

11 Comments

Tired of complex workflows to move data from Dropbox to Snowflake? With Openflow, you can set up a secure, scalable pipeline in just a few clicks, no heavy coding required. Learn how in #DataSuperhero Maja Ferle's blog: https://lnkd.in/gic3fCTK 

316

3 Comments