Kuba Nowakowski’s Post

How CQL runs on Accord by Ariel Weisberg Ever wondered how Apache Cassandra is evolving to support strictly serializable, multi-key transactions? Dive deep into the engine room with Accord, the new consensus protocol set to revolutionize how CQL operates. 🚀 https://lnkd.in/dJh8ktRS

Live from #FlinkForward, Yuan Mei shares VERA-X, the first native vectorized engine for Apache Flink. What does this mean? Blazing speed for data processing that supports “I need it RIGHT NOW” use cases (like fraud detection). Her recording will be available soon, but in the meantime, you can learn more in the blog from Ververica | Original creators of Apache Flink® https://lnkd.in/g_wFM64B

🔥 Master the power behind Apache DolphinScheduler’s Worker Load Balancing! Discover 3 key algorithms — Weighted Random, Smooth Round-Robin, and Linear Load (default) — plus weight tuning, warm-up logic, and full technical insights. #ApacheDolphinScheduler #LoadBalancing #BigData #OpenSource 🚀 https://lnkd.in/eG4ah-Fi

An upstream team changed a datatype on a billion-row table this week, and everything downstream broke. No control. No rollback. Just another reminder that most pipelines still assume nothing ever changes. But in reality, change is constant and our systems need to adapt, not crash. The fix isn’t another patch. It’s designing pipelines that evolve safely. Safe promotions mean metadata-only updates, no rewrites. Shadow columns let you isolate breaking changes and backfill when ready. That’s where Apache Iceberg really earns its name stable column IDs, atomic schema evolution, and time-travel rollbacks make change a normal part of the process, not an emergency. Drift will always happen. The question is whether your data platform can handle it. #DataEngineering #ApacheIceberg #SchemaEvolution

I'm excited to speak about Iceberg V3 spec with Tomohiro Tanaka at re:Invent in Las Vegas, Nevada! Join us for "OPN411 | Storage optimization by Apache Iceberg V3's Deletion Vectors". Apache Iceberg evolves at a rapid pace, and one of the recent key enhancements is Apache Iceberg V3 format. Apache Iceberg V3 format introduces Deletion Vectors, allowing you to reduce storage usage while optimizing workloads. This session dives deep into Deletion Vectors: why Deletion Vectors are introduced, how Deletion Vectors are designed, and their contribution in real-world scenarios. You'll learn how to optimize Iceberg table storage with Deletion Vectors in detail. You'll also learn migration strategies from Apache Iceberg V2 format. Your participation and questions will make this an engaging, collaborative learning experience that benefits everyone in attendance. https://lnkd.in/gVGjjGd8 #AWS #Iceberg #BigData #AWSreInvent

You've decided to run a shell script that that collects logs of pod health with your cluster, what should you do first? ✓ Ensure you're connected and authenticated to the correct kubernetes cluster. How can you do this? Please run the command - kubectl config current-context

Data Retention in Apache Iceberg isn’t as simple as setting an S3 lifecycle policy. It’s more nuanced than that. In our latest post, our CTO Yuval Yogev breaks down how retention really works in Iceberg, common pitfalls, and why application-level retention is key for reliable cleanup and compliance 🧹 Start improving retention design → https://lnkd.in/dfRicxuY

Ever feel like Apache Iceberg's perks don’t quite click in your setup? Sometimes it's not the tech — it's the guidance. Luckily, Alex Merced is one of the best at helping data engineers make Iceberg work for them. Today at 5 pm CDT, he's speaking at Confluent's Current on "Tuning the Iceberg: Practical Strategies for Optimizing Tables and Queries." Expect clear, hands-on examples that make even complex tuning feel approachable — no matter what environment you’re running on. More info here: https://hubs.la/Q03QC0XZ0 Not in NOLA? You can still tap into Alex's expertise; his book Architecting an Apache Iceberg Lakehouse is 45% off with code mercedconf25 through December 6th: https://hubs.la/Q03QC3tQ0

New blog post! How Would You Like Your Iceberg Sir? Stream or Batch Ordered? Many teams today are trying to unify their streaming and batch pipelines over Apache Iceberg. But physical unification comes with trade-offs: 🔹 Flink streaming mode and other stream processors want stream-order (temporal order). 🔹 Spark and analytical engines want batch-order (value clustering for pruning). In my latest post, I explore why these layouts are fundamentally at odds and how Apache Fluss and Confluent Tableflow each navigate this tension in very different ways. https://lnkd.in/dwKf8Fua

A very interesting read from Jack Vanlightly. Temporal locality is a fundamental demand of streaming and dealing with the choice of how to work around this for the stream/iceberg interface is a challenge a lot of different platforms are looking at. Makes for a great follow up to several chats I had at Current about tableflow and open table format architecturing choices!