Dimitris Leventeas

Seeing quite a few discussions lately about Kafka/Iceberg integrations being "zero-copy" or not. I think this is largely missing the point. First, where I agree is that this integration should be "zero-effort" for users. Materializing a Kafka topic into an Iceberg table shouldn't require more than a click of a button. Queries should provide a uniform way for accessing the data in both a topic and the corresponding table. This is the stream/table duality, and it should Just Work™. Now, whether this requires to store the bytes of data once in a Kafka topic, and a second time elsewhere for table access, shouldn't really matter from a user perspective. I'd argue storing the data twice is actually a benefit, and in fact it's a pattern well established: it resembles the design of WAL and table files known from databases for decades. I don't think anyone ever complained about this structure in their RDBMS? Which makes sense, it's an implementation detail, opaque to users. But as it turns out, having log and table data separately is even more advantageous for the deconstructed database that is Kafka and Iceberg: you can have multiple readers of the same log (Kafka topic), materializing views in multiple destinations and systems optimized for specific use cases. Maybe multiple Iceberg tables with different projections (think PII), maybe an Iceberg table and a full-text index in Elasticsearch, maybe an... you catch my drift. Furthermore, the log is replayable, so you can recreate views if needed, or you can implement new use cases you didn't originally have in mind. All in all, I think "zero copy" is mostly a red herring. Sure, it can be an optimization for certain scenarios, but mostly it's a distraction from the immense value you get from combining Kafka and Iceberg when done the right (seamless) way.

127

26 Kommentare

P2P systems are amazing and a couple of years back I spent a month going through BitTorrent and its internals (videos on my YT), but this weekend I am digging into something more related but much more fascinating ... I am reading a paper from Google about load balancing in peer-to-peer (P2P) systems which introduces two protocols that refine Chord's consistent hashing model. It improves load distribution while reducing query times and minimal data movement. You will have a great time if you like distributed hash tables or are even fascinated by them. The approaches fix inefficiencies in existing DHTs and offer a much simpler and more practical way to maintain balance dynamically. I have skimmed it once and found it super interesting. I would recommend you to watch my video on Kademlia before digging deeper, but still no hard pre-requisite. This weekend is going to be super interesting :) You can always find the papers I read and the notes (of some) on my paper shelf and other write-ups here arpitbhayani.me/papershelf arpitbhayani.me/blogs I will cover some of these topics in future videos. But until then, you can watch detailed explainer videos I made around system design, databases, and concurrency. youtube.com/c/ArpitBhayani #AsliEngineering #SystemDesign

229

4 Kommentare

Now that the craziness of #MSBuild is over, I want to acknowledge the incredible collaborators who participated in the early development of NLWeb, Microsoft’s open project designed to simplify the creation of natural interfaces for websites. We had collaborators for models, collaborators for retrieval/vector databases, and collaborators using NLWeb on their websites. It is pretty amazing to see what they were able to accomplish very quickly.    In the below video reel, you can see how NLWeb transforms websites into AI-powered apps, with deeper semantic understanding. Additionally, you can learn more about each partner and their experience with NLWeb in the pioneer profiles at https://lnkd.in/gDEXMEN5. #NLWeb   • 𝐂𝐨𝐦𝐦𝐨𝐧 𝐒𝐞𝐧𝐬𝐞 𝐌𝐞𝐝𝐢𝐚: Plan the perfect family movie night with age-appropriate content on the topics they are passionate about, or find books for a kid who is struggling with "not winning" at soccer and needs some stories that demonstrate resilience and hard work. • 𝐃𝐞𝐥𝐢𝐬𝐡 (𝐇𝐞𝐚𝐫𝐬𝐭): Instant recipe matches—“quick vegan dinner with mushrooms and pasta”—tailored to your tastes. https://lnkd.in/gvijHK_p  • 𝐄𝐯𝐞𝐧𝐭𝐛𝐫𝐢𝐭𝐞: Discover events by intent, not keywords. https://lnkd.in/gYh3MiYq • 𝐈𝐧𝐜𝐞𝐩𝐭𝐢𝐨𝐧: Sub-second conversational queries using diffusion LLMs. https://lnkd.in/guPf33wb • 𝐌𝐢𝐥𝐯𝐮𝐬: Vector database support for retrieval.   • 𝐎’𝐑𝐞𝐢𝐥𝐥𝐲 𝐌𝐞𝐝𝐢𝐚: Queryable technical library, like chatting with a resident expert. https://lnkd.in/gvVsHAt9 • 𝐐𝐝𝐫𝐚𝐧𝐭: Lightning-fast, intent-aware search via its vector database. https://lnkd.in/gnR3i5XU • 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞: Leverage both hosted models and Cortex search for retrieval. • 𝐓𝐫𝐢𝐩𝐚𝐝𝐯𝐢𝐬𝐨𝐫: Conversational travel planning, from “Where should I go this fall with kids?” to full itineraries in one go. https://lnkd.in/gV3hiDnh You can check out the NLWeb code repository at https://lnkd.in/ghrNBsp2.    

419

19 Kommentare

AirBnB recently did a large scale migration of code using LLMs. I've been trying to find good datapoints for things like this, so personally reading this blogpost (link in comments) was quite exciting. I have done a crapton of these migrations by hand (aided with static analysis/codemod scripts in the past) and I'm excited to see where LLMs can go in this space. Some key takeaways from this post (which resonated well for me, similar to how I'd do it as a human funnily enough): * Break the migration into manageable chunks: * Migrate each file independently * Within that too, have a logical set of sequential steps. * Don't try to do the whole thing in one shot * Choosing the exact specific context matters a lot for the LLM * Retries help -- if it fails, just try again * It's easy to get "good enough" to e.g get 75% of the way there * From there to 97% took a lot more work * ... and it doesn't make sense to go to 100% (in terms of ROI) Money quote from the article below: > Airbnb recently completed our first large-scale, LLM-driven code migration, updating nearly 3.5K React component test files from Enzyme to use React Testing Library (RTL) instead. We’d originally estimated this would take 1.5 years of engineering time to do by hand, but — using a combination of frontier models and robust automation — we finished the entire migration in just 6 weeks. Have you done an large scale migration (LLM or otherwise)? If so, I'd love to chat. Looking to learn from experts in the field.

210

10 Kommentare

How to attend a tech conference or summit like a boss. Recently one of our software engineers in Zurich visited Waterloo for a Google conference. Mircea Peia’s activities during his two week stay serve as a model for anyone attending a technical conference or summit. Here’s an overview of his activities: - Attending all relevant conference sessions, breakouts and ice breakers - Connecting with senior technical leaders at the conference to discuss challenges faced by out team and our SRE org to seek solutions, advice and new perspectives - Landing a mentorship opportunity with a tenured L8 UTL for career and technical advancement - Putting together a timely trip report and presenting it to our team with learnings, key takeaways and action items - Delivering a talk on the project track he is working on, reporting on the wins and painting an image of his vision and where he’d like our team and the intern he is mentoring to go from here - Having a 1:1 meeting with all members of our team as well as local managers or leads from neighbouring teams - Joining our team’s social outing - Managing to squeeze in time to visit Toronto and Niagara Falls to do touristy things on his Canadian trip Learning, networking, making valuable contributions, establishing new connections and strengthening existing ones, paving the road for future growth, socializing and having fun. That’s how you attend a conference like a boss!

26

New Rust blog post on why we need (at least) ergonomic, explicit handle creation: https://lnkd.in/eDVBEhzu > **This blog argues that we should make it ergonomic to be explicit**. This wasn't always my position, but after an impactful conversation with Josh Triplett, I've come around. I think it aligns with what I once called the soul of Rust: we want to be ergonomic, yes, but we want to be **ergonomic while giving control**. I also like Tyler Mandry's *Clarity of purpose* contruction, *"Great code brings only the important characteristics of your application to your attention"*. The key point is that *there is great code in which cloning and handles are important characteristics*, so we need to make that code possible to express nicely. This is particularly true since Rust is one of the very few languages that really targets that kind of low-level, foundational code.

41

At Netlify, we've quietly released something that I’m really proud of. With a single npm package, you can now run the full Netlify platform right in your local environment, with a minimalist API built on web standards. https://lnkd.in/d8YP82aq Here’s why I'm excited about it. Being able to emulate the full production environment when developing locally has always been a key component of the Netlify experience. You (and your agents) should be able to leverage all of our primitives as the application is built, with the shortest possible feedback loop. Historically, this capability lived in the Netlify CLI through commands like `netlify dev`. And while the CLI is still an incredibly important entry point into our platform that we'll continue to develop, a lot has changed in the ecosystem since we first launched it a decade ago. Tools like Vite have unlocked a rich ecosystem of web frameworks with powerful local development flows, with great developer experience. If you’re building a site on Astro, the most natural entry point into local development is `astro dev`. Same goes for TanStack, Nuxt, etc. So rather than asking developers to abandon the flow their framework encourages, we need to weave ourselves into that experience. We wanted to bring Functions, Edge Functions, Blobs, Cache API, Image CDN, redirects, environment variables and more right into that flow. In order to do this, we first had to rebuild the local development flow for each one of these primitives, decoupling them from the CLI codebase and structuring them as self-contained packages that conform to the Fetch API. Here's the result: https://lnkd.in/dZ43H3TB (This took some effort.) We could then build the `@netlify/dev` module, which takes all the individual primitive packages and composes them into a representation of the Netlify request chain (https://lnkd.in/dxuVqGwj). With that solid foundation in place, we were able to create a Vite plugin that brings the full power of Netlify to Vite applications with a little more than 100 lines of code: - https://lnkd.in/dYq3FqYA - https://lnkd.in/dqe4-Wje - https://lnkd.in/dxAymUQx Last week, we used the same engine to power a new Nuxt module, which brings all of Netlify to `nuxt dev`. - https://lnkd.in/dYq3FqYA - https://lnkd.in/dGiJKY4h I'm really happy with the improved experience these releases have brought to folks building with these tools — but I'm even more excited about what they unlock. Building custom integrations with Netlify is now much more accessible. I can't wait to see what y'all will build. 🔚

74

6 Kommentare

Lessons from the Trenches: What I've Learned as a Principal Engineer in Amazon Search Amazon's [Principal Engineer tenets](https://lnkd.in/eHtuzWMA) provide valuable guidance that comes alive when applied to specific challenges. Let me share how three experiences taught me what these principles mean for me in practice. Early at Amazon, I noticed our search and catalog systems weren't speaking the same language. Search was built for shoppers while catalog served sellers, creating a disconnect in how we understood products. When customers searched, we struggled to connect their intent with the right items. "Technical Fearlessness" meant proposing an overhaul of data flow between these systems rather than continuing with incremental fixes. This required questioning established patterns across multiple organizations. "Leading with Empathy" became essential as teams brought different perspectives. I discovered even basic terms meant different things to different groups. By actively listening and rephrasing in my words—"So what you're saying is XYZ?"—I built bridges between viewpoints. This wasn't just about being nice; it created the shared understanding necessary for technical progress. Another experience taught me about being "Balanced and Pragmatic." After analyzing tens of millions of search queries to understand the filters that customers encountered, I found quality issues invisible in our averaged metrics and aggregated dashboards. We developed fixes but faced a choice: wait for a sustainable solution or deliver immediate improvements. We chose customer experience, rolling out enhancements while confirming their value through testing, then building sustainability afterward. Sometimes the best technical decision isn't the most elegant—it's the one serving customers now while creating space to build properly for the future. Finally, "Learn, Educate, Advocate" took on new meaning with AI's evolution. Realizing I was behind on AI coding tools, I jumped directly into practice—progressing from basic prompts to Q CLI. This led to building a server in one-tenth the usual time, revealing how we might boost productivity across our engineering work. These experiences showed me that Amazon's PE tenets gain meaning through application—practical guides that help navigate complex technical challenges while focusing on delivering better experiences for customers.

88

2 Kommentare

AI promises speed. But are we just trading time now for cleanup later? Here are some things that have been bugging me lately. 🔹 Sure, it writes code in seconds. 👉 But now you’re debugging for hours. 🔹 It produces a prototype quickly. 👉 But it’s fragile, undocumented, and impossible to scale. 🔹 Copilot feels magical. 👉 Until you forget how to design without it. 🔹 AutoML is quick for non-experts. 👉 But they can’t explain what the model’s doing. 🔹 AI scales content effortlessly. 👉 But now your brand voice gets lost in the noise. 🔹 It floods you with alerts. 👉 Then trains your team to ignore them. 🔹 It makes decisions seem simple. 👉 But it can weaken user understanding and control. The product triangle still applies. ⚡ Speed, quality, and cost. But you still only get two. AI speeds up one side. But you still pay for the other two. Use AI, but remember to question it. #AITradeoffs #AIEfficiency #AIProductTriangle #AISpeedQualityCost #AIProductivity

30

23 Kommentare