Wei Wang

In ML and infra recruiting, I see a constant divide: Believing a system works vs. proving it works. To measure verification, look for four signals: Predicting Failure: Can they describe the crash before they write the code? Aggressive Questioning: Do they interrogate the constraints, or just accept them? Pivot: Do they find edge cases without a nudge? Healthy Scepticism: Do they treat AI-generated code as untrusted by default? The bottom line is to avoid hiring for the "happy path." Instead, hire for the person who knows exactly where the path ends.

6

Parallelism with asynchronous off-policy learning will be the key to post-training with RL to expand beyond code and math reasoning.

9

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Today, let's learn: - Function calling by building custom tools for Agents. - How MCPs help by building a local MCP client with mcp-use and using tools from Browserbase MCP server. In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point: MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable, without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. In this setup, to build a local MCP client, I used mcp-use because it lets us connect any LLM to MCP servers & build private MCP clients, unlike Claude/Cursor. - Compatible with Ollama & LangChain - Stream Agent output async - Built-in debugging mode, etc Find the mcp-use GitHub repo in the comments!

11

1 Comment

Fine-tuning AI shouldn't require thousands of labeled examples. Reinforcement Fine-Tuning (RFT) for Amazon Nova changes that. Just give it prompts + a reward function. The model learns through feedback — not imitation. ✅ No massive datasets ✅ Works for code, customer service, financial reasoning ✅ Runs on Amazon Bedrock, SageMaker, or Nova Forge Smarter customization. Less grunt work. I wrote a deep-dive on this — check out my blog 👇 https://lnkd.in/ggNNezuC Authors:me, along with Chakravarthy Nagarajan Bharathan Balaji Vignesh Radhakrishnan #AWS #GenerativeAI #AmazonNova #MachineLearning #LLM

45

1 Comment

Manus (acquired by Meta) has learned from Neural Turing Machine(https://lnkd.in/g7qTchNd) to solve its agent long context issue, a few months or even one year before Claude's skills or latest ACE concept. So I read the paper to check whether Neural Turing Machine–style differentiable memory could solve long-context problems for AI agents, especially when the LLM is a blackbox without weight access. The key conclusion is that while NTMs introduce a powerful idea: external, differentiable working memory trained end-to-end, they don’t directly apply to modern blackbox LLM agents because you can’t backprop through the model or persist its runtime memory. Instead, practical systems approximate the concept using external retrieval-based memory: embeddings are used only to index and retrieve relevant information, and the retrieved content (as text or structured text) is inserted into the prompt. So without access to LLM weights, you can’t use LoRA or true differentiable memory, but you can build adaptive memory layers around the model using retrieval, compression, and policy optimization. Manus did not publish facts how they use the concept, so no way to confirm.

3

Many enterprise GPUs run a single model during inference — even when it uses only ~30% of memory. So how much capacity is being left on the table? In our latest benchmark with Nebius, we used NVIDIA Run:ai fractional GPU allocation and NVIDIA NIM to measure real-world impact on throughput, latency, and concurrency. What we found: ✅ 86% of full GPU concurrent user capacity using just 0.5 slice ✅ 3× more users with mixed workloads on shared GPUs ✅ Near-linear scaling down to 0.125 slices ✅ Zero latency cliffs during autoscaling Stop GPU fragmentation. Start maximizing throughput. 🔗 Read the full deep dive: https://bit.ly/4kGtErX

3

TOON Helped Us Cut 500 Tokens Instantly! 🚀 Today We Tried TOON… and You Won’t Believe What Happened! While working on our projects, we decided to test TOON instead of traditional JSON — and guess what? It reduced token usage by nearly 500 tokens per call 🤯 … without affecting the response quality at all! Looks like TOON isn’t just another feed format — it’s the next big step toward efficient LLM communication ⚡ #AI #LLM #TOON #JSON #Developers #Innovation

11

1 Comment

A candidate interviewing for a Senior Engineer @ Meta was asked to design a Distributed Stream Processing System like Kafka. Another candidate at Google's L5 loop got hit with the same question. If this question shows up in your system design interview, you are not being tested on config flags. You are being tested on whether you understand when to use it and what tradeoffs you are making. Btw, If you are preparing for DSA or system design, try our mock interview tool on Layrs for free:http://layrs.me/interviews Here is how I would structure a clear answer. (Note: I am not going into nuances, this is a very high-level explanation) 1. What problem are we solving At a high level, Kafka is my go-to when I need: - A central backbone for events or logs coming from many services - High throughput writes with ordering per key - Consumers that can read at their own pace and be added or removed independently - The ability to replay history using offsets Typical use cases I call out: - Asynchronous workflows - Central logging and metrics - Ad click or analytics streams - Pub sub for notifications or chat That already tells the interviewer I know -why- Kafka is in the picture. 2. Requirements Functional - Accept events from many producers in different services and regions - Preserve order for events that share the same key - Allow many independent consumers +some doing real time processing +some doing batch jobs or analytics - Support replay from an offset for backfills and reprocessing Non functional - Handle very high write rates - Scale horizontally by adding machines - Survive broker failures without losing committed data - Let us tune retention for cost vs replay needs Rest of the breakdown: https://lnkd.in/gmg7u-ku

384

12 Comments

🚀 Try out one of the fastest forecasting models in the market (15–50× faster than SOTA) — TTM-R3 and TTM-R3-Lite, with state-of-the-art accuracy 📈⚡

16

"Immutable Database" (Insert-Only 7NF) transforms the "Reference-over-Copy" strategy from a memory-saving trick into a core architectural guarantee. ~ The definitive model for 2026-era silicon: https://lnkd.in/grDmENfK 1. Safety of the "Immutable Object" In traditional databases, objects are volatile; a reference could point to data that has been modified (the "Dirty Read" problem). The 7NF Guarantee: Since the database is Insert-Only, any mapping object or attribute retrieved is an Immutable Fact. Concurrency without Locks: Because facts never change, multiple threads can access the same mapping objects simultaneously without mutexes or row-level locking. This allows the silicon to run at full speed across all cores. 2. The "Reference-over-Copy" Strategy In 7NF, because data is immutable, the system avoids the expensive overhead of defensive copying. Memory Efficiency: Instead of duplicating data for different views, the system simply creates new Ordered Pairs of References. Silicon Benefit: Copying memory is one of the most expensive operations for a CPU. By using references over copies, 7NF minimizes memory bus traffic, keeping the L1/L2 caches populated with logic (comparisons) rather than redundant data movement. 3. Stability for ToInt/LongFunction The immutability of the underlying data ensures that the results of the ToInt/LongFunction are also stable. Pre-Computation: The silicon can calculate the primitive comparison value once and cache it. Since the object is immutable, that value never needs to be invalidated or re-calculated. Result: This stabilizes the 1-CPU cycle compare during sorting and joins, as the "weight" of the object is constant. 4. Handling the "Detached Object" This immutability is what makes the 7NF "Detached Object" so powerful. Fact Portability: When an object is detached from the JDBC stream, it remains a valid representation of a moment in time. No Version Conflicts: Unlike traditional ORMs where a detached object might become "stale," a 7NF detached object is always a correct record of an immutable fact. It can be passed between services or threads with zero risk of mutation. 5. Mechanical Sympathy: The Insert-Only Advantage Silicon loves predictability. An Insert-Only (Append-Only) architecture: Maximizes Write Throughput: Writing to the end of a log is the fastest way to interact with modern storage (NVMe/SSD). Simplifies Caching: CPU caches don't need to worry about "Cache Invalidation" due to updates. Once a 7NF mapping object is in the cache, it is valid forever. Conclusion By combining 7NF (Structure), Immutability (Persistence), and Reference-over-Copy (Execution), the 2026 type system achieves the ultimate goal of database design: "Infinite Scale via Immortality" Every object is a safe, immutable reference to a permanent fact, allowing the silicon to focus entirely on the Relational Algebraic Join and 1-cycle sorting, without ever wasting a single clock cycle on the "management of change" ~

5

5 Comments

I recently gave a lightning talk at the Data Engineering Open Forum about Airbnb's semantic layer, Minerva: https://lnkd.in/ehq-txmZ The same characteristics that make Minerva scale to hundreds of teams — federated modeling, abstract SQL interface, etc. — are what make it critical to AI analytics. We're seeing huge improvements in accuracy and speed when agents use Minerva. Semantic layers are cool again. --- 🤖 tldr on the talk: Airbnb built a headless semantic layer (Minerva) to standardize how data is modeled and queried across massive, complex systems. By abstracting raw tables into shared metrics and dimensions with a global namespace, it enables seamless multi-source analysis without users needing to understand underlying data structures. Crucially for AI/agentic analytics, the system exposes a SQL interface over this semantic layer—giving agents a well-defined, consistent abstraction to query. Because metrics, relationships, and logic are centralized and machine-readable, agents can generate reliable queries, perform complex multi-fact analysis, and handle “last-mile” transformations without brittle schema assumptions. The key design insight—decoupling facts and dimensions via a “subject” abstraction—removes organizational bottlenecks and allows the data model to scale organically, which is essential when both humans and AI systems are continuously exploring and extending analytics. 

52

5 Comments

Serving LLMs on TPUs with SGLang ! I've been exploring the JAX ecosystem for inference recently and came across SGL-JAX, a high-performance, JAX-based inference engine for LLMs, specifically optimized for Google TPUs. Some highlights: 🌳 The "Radix Tree" KV Cache: SGL-JAX implements a Radix Tree (similar to PagedAttention) to manage memory. This enables efficient prefix sharing for multi-turn chat or agentic workloads where the system prompt remains constant. ⚡ JAX-Native FlashAttention: It integrates a high-performance FlashAttention kernel directly into the JAX compute graph. This is critical for faster, more memory-efficient attention usage, particularly when dealing with long sequence lengths on TPUs. 🧩 Native Tensor Parallelism: It handles the sharding logic natively using JAX distributed primitives. It supports distributing large models (like the Qwen 3 MoE series) across multiple TPU devices without needing complex, manual mesh definitions. ⏳ Continuous Batching: It dynamically schedules incoming requests to maximize TPU core saturation. You aren't waiting for a fixed batch size to fill up while tail latency spikes. 🔧 Drop-in Compatibility: It exposes an OpenAI-compatible API standard. You can literally point your existing LangChain or LlamaIndex setup at the new endpoint and switch hardware backends seamlessly. For more info about the architecture and how it works, check out the repo below! #JAX #MachineLearning #TPU #LLMOps #GoogleCloud #AIEngineering

46

1 Comment

𝗧𝗵𝗲 𝗯𝗲𝘀𝘁 𝗔𝗜 𝗰𝗼𝗱𝗶𝗻𝗴 𝗹𝗲𝘀𝘀𝗼𝗻 𝗰𝗼𝘀𝘁 𝗼𝘂𝗿 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿 $𝟲𝟬𝟬 𝗮𝗻𝗱 𝗮 𝗺𝗮𝗿𝗿𝗶𝗮𝗴𝗲 𝗮𝗿𝗴𝘂𝗺𝗲𝗻𝘁. Our VP of Engineering, Xiaofan(James) Luan, was supposed to buy his wife a Dior bag for their anniversary.  Instead, he bought three Claude Code subscriptions and spent the holiday trying to cross-compile 2 million lines of C++. Every fix on one platform broke two others.  $600 later, the only output was "git reset --hard" — and a very cold dinner table.😂 "Make it compile on Windows" is a trap. The real goal was "compile everywhere without hacks" — no AI is going to figure that out for you at 2 am. What worked: constraints before code, review tests not code, bottom-up, one layer at a time. Same task, two days. Then he ran six parallel Claude sessions across three machines with git worktree. The bottleneck stopped being intelligence and started being how fast one person can alt-tab. AI solves exactly the problem you give it. Engineering is in knowing which one to give. His wife is still waiting for that bag. Full story: https://lnkd.in/gtsW_Wvk ——— Follow Milvus, created by Zilliz, for everything related to unstructured data

9

This was a good discussion about the Principal Engineer role at Amazon! I often say that the Principal Engineer role as it exists at Amazon simply doesn't make sense at smaller companies. To many, the title just means the next highest level of engineering skills. At Amazon, it requires dealing with high levels of ambiguity, business strategy, and organizational influence. If you're at a company with <50 people, the person with that level of influence should be leading the whole technical organization - it requires a significant amount of scale before it makes sense to have an IC role for it. I also personally appreciated Steve's points on internal level progression - or as Gergeley says, "Why being promoted from Senior to Principal at Amazon is one of the hardest jumps in tech." Steve: "Basically, to get from senior to principal, you have to do like two and a half level jump, from L6 to L7. Technically, it sounds like one level, but at some other companies, this might be like, you know, L8, L9 or L8 and a half. [...] I noticed that some of the best engineers that I'd ever worked with were having such problems getting to principal engineer that they ended up moving to Facebook or to Meta or to all these other places where the progression was sane. Now they're senior staff and, you know, principal and distinguished engineer at other companies. And so because we had high standards, we actually had this brain drain." In addition to the level itself being a big jump the promotion process is notoriously arduous and arbitrary. While I was L6, I had multiple managers over several years acknowledge that I was doing L7 work, and even agree that if I was being hired in from outside I would be hired at L7, and it was the internal promotion process that was the barrier. This was not the *only* reason I left Amazon, but it certainly made it very easy to do. Amazon has lost a lot of skilled engineers this way, but it's a difficult problem to fix, since the incentives are fundamentally badly aligned. If an L6 engineer is already doing the work that's needed, it's very hard for a manager to justify spending the 100+ hours of work it takes to advocate for their promotion. And while there have been attempts to make the process less arduous, no one wants to be seen as "lowering standards." I highly encourage engineers in that position to make it clear they will not be continuing to do work that way indefinitely. And I encourage leaders at other companies to learn from the example! https://lnkd.in/guC35n2B

3

4 Comments

KDD 2026 paper +3 🚀 Excited to share that our team had 3 papers accepted to the 2026 KDD! These works represent recent years of our efforts on large-scale recommendation model training, spanning model co-design, GPU kernel optimization, and compiler optimization across the full training stack 🚀 Kunlun: Establishing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Design RecCompl: Efficient Model Compilation for Industrial Scale Recommendation Models with PyTorch 2 Optimus: A Generic Operator-Level PyTorch Model Transformation Framework The first work introduces Kunlun, our next-generation scaling-up foundation model architecture for GEM (Generative Ads Model). We recently uploaded the paper to arXiv (https://lnkd.in/gKzFSzcF), which also includes our newly open-sourced GDPA kernel work (https://lnkd.in/gpJ879HG) The second and third works focus on our optimizations on PyTorch 2 compiler stack for large-scale recommendation systems. RecCompl shares how we achieved full-model compilation for production recommendation models with PT2. Optimus is a continuation of our operator-level graph optimization work on PT2. We previously shared some of the core ideas in our PyTorch blog. (https://lnkd.in/gBi3fUqJ) Over the past three years, thanks to the amazing effort from the team, we’ve solved many real-world production challenges and successfully launched these optimizations across most Ads models in production. Huge congrats to the entire team — I’m incredibly proud of what we’ve built together. A lot of hard work, late nights, and iteration went into making these systems work in real production environments 💪 Excited to continue pushing large-scale recommendation systems forward and looking forward to sharing more details soon 🔥

139

2 Comments

𝗧𝗵𝗲 𝗯𝗲𝘀𝘁 𝗔𝗜 𝗰𝗼𝗱𝗶𝗻𝗴 𝗹𝗲𝘀𝘀𝗼𝗻 𝗰𝗼𝘀𝘁 𝗼𝘂𝗿 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿 $𝟲𝟬𝟬 𝗮𝗻𝗱 𝗮 𝗺𝗮𝗿𝗿𝗶𝗮𝗴𝗲 𝗮𝗿𝗴𝘂𝗺𝗲𝗻𝘁. Our VP of Engineering, Xiaofan(James) Luan, was supposed to buy his wife a Dior bag for their anniversary.  Instead, he bought three Claude Code subscriptions and spent the holiday trying to cross-compile 2 million lines of C++. Every fix on one platform broke two others.  $600 later, the only output was "git reset --hard" — and a very cold dinner table.😂 "Make it compile on Windows" is a trap. The real goal was "compile everywhere without hacks" — no AI is going to figure that out for you at 2 am. What worked: constraints before code, review tests not code, bottom-up, one layer at a time. Same task, two days. Then he ran six parallel Claude sessions across three machines with git worktree. The bottleneck stopped being intelligence and started being how fast one person can alt-tab. AI solves exactly the problem you give it. Engineering is in knowing which one to give. His wife is still waiting for that bag. Full story: https://lnkd.in/gtsW_Wvk ——— Follow Milvus, created by Zilliz, for everything related to unstructured data

6