Reynold Xin

My thoughts on the recent Cerebras & OpenAI partnership Inference speed at scale is a core bottleneck for AI. The partnership between Cerebras and OpenAI signals a shift many haven’t internalized: once models are good enough, latency and throughput under concurrency become the binding constraints, not model quality. At scale, inference is the product. Every user interaction, agent step, and reasoning loop is gated by P99 latency under load, not by how smart the model is in isolation. Milliseconds compound into user experience, developer velocity, and revenue. This isn’t about cheaper inference. It’s about faster inference unlocking new economics: real-time agents, deeper reasoning chains, and high utilization without tail-latency blowups. OpenAI isn’t just securing capacity. They’re prioritizing architectures where time-to-answer is a first-class design variable. When intelligence commoditizes, time becomes the scarce resource. Yuval Rozio David Fabregat Tristan Capes-Davis https://lnkd.in/exfBGZzs

30

For years, the story was: TPUs for unmatched performance, but with a steeper learning curve. The new vLLM TPU backend just rewrote that story. By unifying the PyTorch and JAX ecosystems, we've removed one of the biggest barriers to adoption. Now, the massive vLLM community can tap into the raw power and efficiency of TPUs without the heavy lift. This is more than a performance boost; it's about democratizing access to hyperscale AI infrastructure. A huge step forward for the entire open-source community. #AI #LLM #TPU #GoogleCloud #vLLM #OpenSource #GenAI https://lnkd.in/e6RzQRuV

21

[AI paper] GRPO has a hidden flaw in Multi-Reward settings. It’s time to decouple your normalization. 📉 As AI Architects, we are moving from single-objective RL (just "get the answer right") to multi-objective RL (accuracy + format + length + safety). We typically sum these rewards and throw them into GRPO. A new paper from NVIDIA, "GDPO: Group reward-Decoupled Normalization Policy Optimization," demonstrates that this naive summation causes Reward Signal Collapse. The Architectural Failure Mode: If you sum disparate rewards (e.g., a binary Format reward and a scalar Accuracy reward) and then normalize, different raw reward combinations can map to identical advantage values. Example: A rollout with rewards (0, 2) might yield the same normalized advantage as (0, 1) due to group statistics, effectively deleting the gradient signal for the second objective . The Solution: GDPO The fix is architectural: Decoupled Normalization. Instead of normalizing the sum, GDPO normalizes each reward objective independently within the group before aggregation. Impact: This restores the resolution of the training signal. On Tool Calling and Math Reasoning tasks (DeepSeek-R1/Qwen), GDPO converges where GRPO fails or plateaus. If you are training agents that must balance strict formatting (JSON) with reasoning quality, this is the correct objective function to use. 👇 Link to the paper in the comments. #AIArchitecture #RLHF #NVIDIA #GRPO #DeepLearning #LLM #Alignment #Research

21

1 Comment

xAI is sharing spare GPU capacity with Anthropic. Their cluster utilization was reportedly running at 11%. The industry average isn't much better — 40%. This isn't an xAI problem. It's a structural one. Most clusters are reserved by a single tenant. But compute demand from any one team is inherently uneven: quiet during planning cycles, maxed out before a model deadline. The gap between those two states gets eaten by idle GPUs no one is using, while teams without reservations queue for hours to get a single node. Velda is making compute fluid. If you reserve GPUs with us, your unused capacity doesn't go to waste: it converts automatically into spot credits you can draw on later. You will have prioritized access to the spot pool, which comes from unused compute of reservations that would be otherwise wasted. Velda allows you to run GPU jobs with a simple command prefix; Preempted jobs retry automatically from last check-point. Getting cluster access should feel like using your workstation, no manifest, no extra setup. And we're not going to mark up the hardware. You pay what the GPU provider charges, no platform tax on top. If you have an existing provider relationship, we'll work with them directly. The capacity is there. It just needs to move.

23

6 Comments

Proud to share our latest work on PARSE — Instacart’s self-serve platform for product attribute extraction using multi-modal LLMs (text + image). From identifying “80 sheets” on packaging to reasoning over “3 boxes of 124 tissues,” PARSE brings accuracy, coverage, and speed to our catalog systems at scale. Key highlights: - Multi-modal LLMs for robust attribute extraction - Zero/few-shot setup with confidence scoring - 70% cost savings on simpler attributes - Built-in quality review, versioning, and ongoing automation Huge kudos to the entire catalog team and our cross-functional partners 👏 .

35

Presented at the ML+X community meeting at UW-Madison today on deploying a retrieval-augmented generation (RAG) system on AWS Bedrock and evaluating LLM performance at scale. Our team took KohakuRAG, the #1 solution from the 2025 WSDM WattBot Challenge, and deployed it as a fully serverless pipeline on AWS Bedrock. I built the Bedrock integration, evaluation framework, and cost tracking infrastructure, then benchmarked 9 LLMs and 3 ensemble strategies across 282 questions to answer a practical question: which model should you actually deploy in production? A few results that stood out... - Ensemble majority voting (0.840) outperformed every individual model. - Llama 4 Maverick delivered 98% of the top score at a fraction of the cost and latency. - Model behavior matters as much as raw accuracy; our highest-citation model finished last overall due to aggressive refusal behavior. - Text-only embeddings create a ceiling on figure-based questions that no LLM can overcome. The full recording will be available on ML+X Nexus: (https://lnkd.in/gSTe_EdW) soon. I would definitely recommend watching the whole thing if you are interested in deploying or learning more about RAG applications! Update: Now live at https://lnkd.in/gv7_jHkv Thank you to Christopher Endemann for the mentorship and for bringing me onto this project; I learned more about applied ML infrastructure than I have in any course in just a few months. Thank you to Blaise Manga Enuh, Ph.D. for the collaboration on the local deployment pipeline.

21

4 Comments

Agents don’t flow — they live. I keep seeing new “agent” frameworks that simply repackage workflows — connecting steps in a DAG (Directed Acyclic Graph) and calling it autonomy. The latest example: OpenAI’s AgentKit announced at Dev Day → https://lnkd.in/g7FCW6Mt But autonomy isn’t something you add to a workflow — it’s something that defines an agent. True agents don’t flow from one step to the next. They exist in environments — perceiving, deciding, and acting continuously based on what they know, sense, and aim to achieve. When we treat agents as workflow nodes, we lose what makes them agentic in the first place: persistence, specialization, and emergent coordination. Sustainable architectures won’t come from orchestrated flows dressed up as agents. They’ll come from systems where intelligence arises from interaction, not instruction. (Not a critique of any vendor — just a reminder that autonomy isn’t a feature you wire in; it’s the nature of an agent itself.) #AI #Agents #SystemsArchitecture #Autonomy #MultiAgentSystems #LLM #EmergentIntelligence #AgenticAI #Innovation #DesignPhilosophy

59

My Take on “Agentic” Frameworks and Real AI Engineering: There’s a lot of excitement around “agentic” frameworks, OpenAI’s AgentKit, LangChain, LlamaIndex, and others, all promising to simplify how we build intelligent systems. They’re great for quick prototypes, but when you’re designing production AI products that have to last, the picture changes. To me, an agent isn’t a visual node or a black-box workflow, it’s simply a modular, testable class that performs a defined task using an LLM or transformer model under strict control. In other words, it’s just good software design: clear inputs and outputs, deterministic orchestration, and measurable results. I prefer to build hybrid systems that combine structured engineering with state-of-the-art NLP and transformer architectures, using the LLM as a controlled component, not the entire system. That keeps the logic transparent, debuggable, and adaptable as models and requirements evolve. Frameworks like AgentKit can still help at the edges, for evaluation, UI, or quick orchestration, but the core intelligence and logic should always live in code you own and understand. That’s how you build AI that’s not only powerful today, but maintainable tomorrow. #AI #MachineLearning #LLM #SoftwareEngineering #NLP #ProductDesign

49

4 Comments

The GPU budget isn't what's killing your AI ROI. It's the silent compute burn happening before the model even touches the data. 🛑 Everyone is obsessing over the cost of model training, but here is the unpopular engineering truth: for most enterprise AI projects, data preprocessing consumes significantly more compute than actual model training. Heavy workloads like deduplication, normalization, and label cleaning are massive cost centers. Yet, while engineering teams meticulously track GPU hours, almost nobody instruments their data pipelines. You cannot optimize what you do not measure. If you lack granular visibility into your preprocessing compute, you are flying completely blind on a cost center that can easily exceed your entire training budget. Stop bleeding compute in the dark. Instrument your data pipeline. Take control of your AI unit economics today at https://zurl.co/bieZo. 👇 #DataEngineering #FinOps #EnterpriseAI

6

Wondering when this was coming. We've been using AWS Bedrock for all Anthropic models since the beginning. Lower latency. More secure. Favorable cost. Also just recognizing they don't have the infra (yet) to compete at the scale they need. Also also the fact that they are saying, "yeah sure we don't care...run our models from anywhere and we'll build the tools that know how to use them best" is brilliant.

5

5 Comments

Lovable hit $100 million ARR 8 months from launch. Anysphere's Cursor took 12 months to hit $100 million Before that, It took Wiz 18 months to reach $100M And Deel 20 months Ramp 24 months All these companies exhibited #escape #velocity. If you look at how they conducted business, these companies focused on #distribution early on in the company's journey. "The game of #startups is if the startup can gain #distribution faster than the incumbent gain #innovation ", as Alex Rampell wrote in his 2015 blog post Alex Rampell in his blog post “Distribution vs Innovation” As Lovable's stats demonstrate, the escape velocity is becoming larger and more aggressive, particularly with #AI. Will we see companies reaching $100M ARR in 6 months? Will we see solopreneur companies achieving the same escape velocity? The question for other #startups, those that funded but not achieving quite the same escape velocity -- what are you doing to steer yourself onto the escape velocity path? As Brian Balfour wrote in this excellent blog: "The Big Squeeze: Why Escape Velocity Is More Important Than Ever"-- #Distribution is the seed that unlocks harvest (I recommend every startup founder read this blog, link in the comments) - Who can you integrate with? - Which channels can you attain distribution faster than others? - Who should you partner with to give you that market #uplift? #distribution is indeed more important than ever. If you are a startup and you are not asking these questions from day 1, you will be left behind.

62

13 Comments

You don't need a separate reference model for RL. We built an RL pipeline built on a multi-turn-focused modification of Self-Distillation Policy Optimization (SDPO) that uses the same model in two roles: student and teacher. The difference? Context window. → The student generates actions with no knowledge of the outcome. The teacher re-evaluates the same tokens with hindsight: the observation from each action, terminal test output, and a successful rollout from the same prompt. The per-token advantage is just the logit gap between these two views. → This gives you dense, per-token credit assignment instead of a single scalar reward for the whole trajectory. Think tokens get 0.25× weight (loosely guided, not tightly constrained). Action tokens get full gradient. System/user/tool result tokens get nothing. → The "teacher" is not a frozen checkpoint or a separate reward model. It's your current policy, conditioned on richer information. This is well-defined under MoE because same parameters with different routing is normal operating behavior. → We use a tiered reward system with hard gates only for truly unrecoverable failures: sandbox escapes, test corruption, harness crashes, timeouts, infinite loops. Everything else like wrong directory, failed commands, incorrect approaches are learnable signal, not wasted compute. The key insight: most of the information you need for credit assignment is already inside the model. You just need to ask it the right question with the right context. On LiveCodeBench v6 SDPO reached 48.8% vs GRPO's 41.2% beating Claude Sonnet 4 with an 8B model! That's bonkers.

5

Quite an interesting (though relatively short) article on building vector search systems in real-world production settings. It describes an infrastructure operating at scale, around 100B vectors (200 TiB of vector data), with a p99 latency target of ~200ms and a relatively modest ~1K QPS, which still makes the system highly non-trivial. What stands out is the range of engineering techniques required to make this work in practice, including hierarchical clustering, binary quantization, multi-stage ANN pipelines, and other aggressive compression and pruning strategies (We,p eople who build search engines know very well how much engineering effort these systems require, and at the same time how substantial the advantages can be once they are working at scale.). The system design is clearly driven by the constraints of scale, where every optimization matters. More broadly, it reflects a wider 2026 trend, large-scale vector search is becoming a core infrastructure primitive. While earlier systems focused mostly on RAG over relatively small or curated corpora, today’s deployments operate at tens or even hundreds of billions of vectors, with real-world latency requirements (often ~200ms p99) and increasingly high throughput demands (from 1K QPS in some systems up to 100K–1M QPS in others). The key takeaway is that ANN systems are now treated as first-class infrastructure, where even single-digit percentage improvements in index size, latency, or throughput translate into significant cost and scalability advantages. https://lnkd.in/eSFcpdPZ

62

1 Comment

AI is moving fast—but the real challenge isn’t adoption, it’s operation. Monitoring, evaluating, and controlling cost and latency are what turn AI from an experiment into something production-ready. The teams that get this right won’t just build AI systems—they’ll run them reliably at scale.

13

There are also a few startups trying to do this :) Figure out how to use GPUs for other distrubuted processing things besides matrix multiplication.

4

#AI coding #agents are changing how I evaluate vendors. A strong argument for products is if they can be configured purely through code and #API / #MCP. Not because they have their own AI features - but because they let your AI agent do the work. Lightdash is a great example. It is not the most mature BI tool, but probably one of the most AI-ready - because they made all data and configuration accessible via API and gave AI agents the context to act on it, so Claude Code can end-to-end create dashboards by modelling data in the data warehouse and then build the visualizations. If your product is not API-first, AI cannot bring in outside context or act autonomously within it. That is becoming a real competitive disadvantage. https://lnkd.in/e6Gp3MGj

53

14 Comments

YellowDog.ai just set a 10x benchmark uplift in scale computing, delivering 40,000 tasks per second (TPS) and managing 100,000 compute nodes in the cloud. What's even more interesting, that's 2x IBM Symphony and opens an intriguing pathway for these until-know closed/captive systems.

11

this is super important because TPUs have definitely advantages for cost and performance over other accelators but were much more difficult to use. no longer the case. "vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path. It is not only faster than the previous generation of vLLM TPU, but also offers broader model coverage and feature support." https://lnkd.in/diDKzYbS

35

1 Comment