Neha Narkhede

A nice article about a great storage engine, of course I have some feedback. WiredTiger might be the perfect storage engine for MongoDB, even if I hope one day that it will be joined by an LSM-based one. It is great for large objects (docs) and really is a modern storage engine -- note that any "modern" engine that is widely used in production will still be many years old -- these things take time. A performance comparison for WiredTiger vs storage engine that does MVCC via latches (not lock-free) would be great content for a conference paper. Similar to "real time", "lock free" doesn't mean (or doesn't guarantee) fast although that is often implied. While "real time" means predictable, "lock free" doesn't even mean that -- see "lock free" vs "wait free" at https://lnkd.in/gT4RUhm2. The article uses "lock" to describe both locks and latches and this will cause confusion. In storage engine land a latch is an implementation detail, often a pthread mutex, used to guard access to in-memory structures and only held for a short time. While a lock is per row, page, table or database and usually held until the end of the statement or transaction. Users think about locks, people working on the engine worry about latches. Pre-MVCC engines like SQL Server, DB2, and Sybase often do (or did) locks per row with escalation to per page and per table. I am not an expert on them and await the "well, actually". MVCC engines don't take read locks unless you explicitly request it (SELECT ... FOR UPDATE), but they will use write locks. I assume pessimistic CC, with the less-popular optimistic CC there won't be write locks and there will be more retries. MyISAM did per-table and mmapv1 did per-database (or per-instance). Neither was good at concurrency. https://lnkd.in/gVgugrxA I didn't understand the section on lock-free updates. * Is the buffer pool organized as a skip-list? * Are pages the entries in the skip-list? * Does any change to a page mean copy-on-write (copy whole page, make your change in the copy, replace old entry with your copy)? For performance, I want to see: * read-only, for working set that does and does not fit in cache * read-write, for working set that does and does not fit in cache For read-write I want to see tests with more and less contention, and by contention I mean updates to the same doc. The high contention workload in the post is append-only (no updates, just add new docs). I also want to see how performant WT is when doing eviction for both clean and dirty pages. High perf storage puts much more stress on eviction algorithms, and eviction isn't trivial.

31

1 Comment

𝗧𝗵𝗶𝘀 𝗶𝘀 𝘄𝗵𝗮𝘁 𝗺𝗼𝗱𝗲𝗿𝗻 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲 𝗹𝗼𝗼𝗸𝘀 𝗹𝗶𝗸𝗲. 𝗔𝗽𝗽𝗿𝗼𝘅𝗶𝗺𝗮𝘁𝗲𝗹𝘆 𝟰𝟴.𝟱% 𝗼𝗳 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀 𝘂𝘀𝗲 𝗢𝗽𝗲𝗻𝗧𝗲𝗹𝗲𝗺𝗲𝘁𝗿𝘆 𝗳𝗼𝗿 𝗼𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆, 𝘄𝗵𝗶𝗹𝗲 𝟮𝟱% 𝗮𝗿𝗲 𝗽𝗹𝗮𝗻𝗻𝗶𝗻𝗴 𝘁𝗼 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗶𝘁 𝘀𝗼𝗼𝗻. Nearly 75%! OpenTelemetry stands out as a prime example of new, successful modern Open Source. It has rapidly gained traction, emerging as the standard for cloud-native observability among vendors, platform engineers, and developers. It is now the de-facto standard and positioned to be the foundation of contemporary observability. It's community boasts nearly 10,000 contributors from around 1,200 companies. I have a favorite - ari zilka (Terracotta, Hortonworks, New Relic Labs) is back with his new Open Source venture with a pretty cool 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 company 𝗠𝘆𝗗𝗲𝗰𝗶𝘀𝗶𝘃𝗲. His usual preciseness, fit and it is really pretty. What I mean is it has this Heroku meets New Relic vibe - that elegance, that simplicity, powered by OpenTelemetry. Cool one to watch. 𝗔𝗻𝗱 𝘀𝗮𝘃𝗶𝗻𝗴 $𝟵𝟲𝟬,𝟬𝟬𝟬𝗸 𝗼𝗻 𝘆𝗼𝘂𝗿 𝗰𝗹𝗼𝘂𝗱 𝗯𝗶𝗹𝗹 𝗼𝘂𝘁 𝗼𝗳 𝘁𝗵𝗲 𝗴𝗮𝘁𝗲. Go check it out: mydecisive.ai.

8

Anthropic just committed $100M to launch the Claude Partner Network. The capital will fund training, certification, and joint GTM for consulting firms and system integrators. They are moving beyond direct API sales to activate a network of software vendors to deploy Claude at scale. Ecosystem building is the new moat in enterprise #AI. The AI war has shifted from training the smartest models to owning the distribution channels.

9

2 Comments

Big moment seeing how NinjaVan scales with TiDB. Their logistics platform powers millions of deliveries every day across six countries, and at that scale, every query, replica, and schema change matters. By moving from hundreds of sharded MySQL instances to TiDB, they unlocked real-time analytics, 44K+ QPS on transactional workloads, and simplified operations so much that two engineers now manage 130TB of data. It’s a great example of what happens when distributed SQL meets real-world scale. Check out how Ninja Van built their next-gen data backbone with TiDB → https://ow.ly/zhKM30sQOvt #TiDB #DistributedSQL #Scalability #RealTimeAnalytics #NinjaVan #HybridWorkloads

14

🚨 How Spring Kafka MANUAL_IMMEDIATE Turned Offset Commits into a Network Bottleneck and Degraded Consumer Throughput Debugged a production issue where a 60K-message burst took nearly an hour to process — not because of broker capacity, but because every acknowledge() triggered a synchronous commitSync(). 🔍 What Actually Happened? Upstream published 60K+ messages in ~10 minutes, while the consumer processed only ~1K messages/minute, creating a backlog that took nearly an hour to drain. The same Apache Kafka cluster handles significantly higher loads elsewhere, so broker capacity was not the issue. The first real clue came from JFR. Consumer threads were RUNNABLE but blocked in network I/O: EPoll.wait ← NetworkClient.poll ← ConsumerCoordinator.commitOffsetsSync ← KafkaConsumer.commitSync ← KafkaMessageListenerContainer.ackImmediate CPU was low, but latency was high. The stack trace made it clear that fetching was not the bottleneck — threads were blocked in the commit path. The consumer container was configured with AckMode.MANUAL_IMMEDIATE. In Spring for Apache Kafka, every acknowledge() in this mode triggers a synchronous commitSync(), blocking until the broker responds. Because the application acknowledged each message individually, every message triggered a full offset commit cycle. So the execution per message looked like: process message → acknowledge() → commitSync() → OffsetCommitRequest to Group Coordinator → Group Coordinator writes to __consumer_offsets → waits for ISR replication acknowledgment → response sent back to consumer → next message ⚠️ Key Insight Kafka fetch is highly efficient — a single fetch request typically returns hundreds of messages, limited by bytes rather than record count. By forcing a synchronous commit per message, we effectively turned a high-throughput pipeline into a network-bound system. We made the control plane too chatty. Instead of: 1 fetch + 1 commit per poll cycle we were doing: 1 fetch + N commits = N + 1 round trips At 60K messages, even ~5 ms per commit adds up to roughly 300 seconds (~5 minutes) of cumulative blocking time per consumer thread. ✅ Fix We switched: AckMode.MANUAL_IMMEDIATE → AckMode.MANUAL With MANUAL, acknowledge() marks offsets locally, and commits are typically deferred until the listener returns. * MANUAL_IMMEDIATE → commits synchronously after each acknowledgment * MANUAL → defers commits, improving throughput but requiring idempotent processing because replays are possible For financial systems, design carefully. 👉 Result After switching to MANUAL, commit frequency dropped from one per message to one per poll cycle, and throughput recovered under the same load. 🧠 Takeaway Fetch = data plane Commit = control plane Don’t let control-plane chattiness choke your data plane. #Kafka #ApacheKafka #SpringKafka #Java #JFR #DistributedSystems #BackendEngineering #PerformanceEngineering #ProductionIncident #Debugging

1

New to Kafka or looking to build your first streaming pipeline? Join us August 5 for a live, hands-on workshop to kickstart your Kafka journey and build real-world streaming use cases with Confluent Cloud.and Apache Kafka®. In this 90-minute interactive session, you’ll: ✅Provision and configure your Confluent Cloud environment, including Kafka clusters ✅Deploy fully-managed source and sink connectors using the UI and CLI ✅Define and enforce schemas using Confluent Schema Registry for compatibility and validation ✅Implement an active-active replication pipeline for real-time data sync between systems https://lnkd.in/gH58YXWB

24

Sharing my latest Medium article on lessons learned from keeping Apache Iceberg tables healthy at scale. In high-churn data systems, table maintenance isn’t optional — it’s operational hygiene. https://lnkd.in/gv2cNUhr

24

The fully managed Couchbase Connector for Confluent Cloud, now available in both Sink and Source configurations. This removes the need to manage connector infrastructure, enabling seamless, bi-directional data movement between Confluent Cloud and Couchbase

1

Just gave this talk at our Berlin Meetup — now online! Decoupled. Compatible. Governed. Why Apache Iceberg REST matters — and how Lakekeeper takes it further. If you’re building modern data platforms, this one’s for you: • Decoupled architectures through metadata-first design • Multi-engine compatibility with the REST Catalog • Governance and access control with Lakekeeper • What’s next for open, interoperable Lakehouses 🎥 Watch the full video: https://lnkd.in/e7–PhkK And if you missed it — we’re back on July 3! Join us for the next Iceberg Meetup → https://lu.ma/pposobem #Iceberg #Lakehouse #DataEngineering #Governance #Vakamo #Lakekeeper #Metadata #DataMesh #OpenStandards

14

Test Coverage is enabled by Open Source Tools like DataKitchen’s DataOps Data Quality TestGen, and Shift Down With DataKitchen’s Open Source DataOps Observability https://hubs.ly/Q03VGLgx0 #dataquality #opensource #dataobservability #dataops #dataengineering

1

K8s KUBERNETES OPTIMIZATION PLAYBOOK Download your free, engineering-first guide to cut waste, improve reliability, and automate optimization (safely). The K8s Optimization Playbook covers: ��� How to assess and eliminate resource risks. ⤷ How to rightsize your pods. ⤷ How to eliminate idle node waste. ⤷ How to use cost-effective instance types. ⤷ When to automate K8s optimization. BONUS: at the end, there’s a 6-step framework + a Production-Ready Optimization Assessment you’ll want to run your cluster through. Download your free copy now! https://hubs.la/Q03Jg1MF0 #kubernetes #devops #k8s #devtool

10

Runway made 5 engineering hires through Paraform - including former Principal Engineers from Twilio and Snowflake.   Runway is a collaborative financial planning platform founded by Siqi Chen and Arya Asemanfar.   Every engineer at Runway works on product that customers use to simulate and make multi-million-dollar decisions. The bar is high by design.   "We aren't looking for volume, we're looking for precision. We need people with real technical depth, strong product instincts, and the kind of ownership mindset that only thrives in a startup environment." - Annie Wu, Head of People & Operations   Traditional agencies couldn't hit their bar. Candidates were either missing the technical depth or weren't ready for a startup environment.   So Runway used Paraform to access specialized engineering recruiters - without scaling internal headcount to do it.   The result: 5 engineering hires, including 2 Staff Engineers now contributing directly to core product, from strengthening modeling infrastructure to supporting new AI-enabled functionality.   "Rather than building internal sourcing capacity from scratch every time hiring spiked, we can dial recruiting support up when needed - and dial it back down when our priorities shift." - Kirill Klimuk, Head of Engineering   They've since expanded their search to Forward Deployed Finance Partners - and already made their first hire there. Read more about Runway's success with Paraform in the comments.

27

1 Comment

Please note. My work on Inference Boosting microservice "InferBoost" is unrelated to any third party websites of similar name . As such dropping the name to refer to it. Renamed Phase 1 work -> https://lnkd.in/daniyxZv By: Ms. Gitanjali GulveSehgal AKA Gigi Sehgal Nee' Gitanjali Gulve Los Altos California

1

Managing Kafka in the cloud just got smarter. This new eBook from Confluent breaks down elastic scaling, infinite storage, and real resiliency - while slashing total cost of ownership. Want to go 10x faster? Download it here: https://lnkd.in/gABvXny4 #Kafka #ConfluentCloud #RealTimeData #TechOps #ConfluentAIDay

3

Deploying LLMs at scale across your organization? LiteLLM (YC W23) and Redis can be a great solution with caching and rate-limiting in Redis built in! check out this article by Rini Vasan to learn more! Tyler Hutcherson Benoit Dion Blair Pierson Ishaan Jaffer

44

1 Comment

People often ask me about the differences in architecture between Amazon Dynamo (the 2007 SOSP paper), DynamoDB (the AWS serverless NoSQL database), and Aurora DSQL (the AWS serverless SQL databases). I captured the key points on durability, high availability, programming model, and trade-offs in my blog: https://lnkd.in/gkMG_tVD TL;DR: Despite the shared name, Dynamo and DynamoDB have little in common architecturally, and make very different trade-off decisions.

632

8 Comments

Today let's talk about Parallel Fan Out/ Concurrent Agentic Design pattern and when parallel agents go rouge What’s the Concurrent Orchestration Pattern? Imagine a scenario where multiple AI agents—each with its own lens or specialty—tackle the same task simultaneously. Instead of a single, step-by-step chain, tasks fan out to different agents in parallel. Then their outputs are merged or aggregated for the final answer. It’s the AI equivalent of a brainstorming session where everyone chips in together. This pattern thrives when you need diverse insights or speed—think ensemble reasoning or reaching a verdict faster. This pattern is commonly used in agentic RAG. Here’s how you can apply it in a Retrieval-Augmented Generation (RAG) system: Step 1: Break the user’s query into smaller sub-queries (e.g., “Define concept X,” “List use cases,” “Give examples”) and map out their dependencies. Step 2: Run those sub-queries in parallel—agents fetch context or process each in isolation (e.g., document retrieval, summarization, external tool usage). Step 3: Once all agents have results, aggregate the findings into a single, coherent response. This technique lets you parallelize independent parts, reducing latency while maintaining clarity—especially effective when sub-queries don’t depend on each other. Super Important : Try to avoid this pattern and use sequential execution even if it's slow, but it might yield a good result, below is the reason As Cognition [the company behind devin] warns in a famous blog post “Don’t Build Multi-Agents”, things can go sideways fast if agents don’t share context or coordination is weak. When agents operate in isolation—making decisions based solely on their own view—the final result might be fragmented, contradictory, or just plain incoherent. Think two agents building different puzzle pieces that don’t fit. The core issues: Context fragmentation: Each agent works in a silo, leading to mismatched assumptions. Implicit decisions: Agents’ outputs reflect unspoken choices that may clash when merged. Coordination complexity: Without strong orchestration, integration becomes error-prone. References : 1. Concurrent Execution Pattern : https://lnkd.in/gZybgm2s 2. LLM Compiler whitepaper : https://lnkd.in/gWwGJhbi 3. Coginition blog Don't build multi agents which shows parallel execution can yield agent drift : https://lnkd.in/gkDKC4TP

14

3 Comments