Christopher Aberger

In this week’s AI infra chat, we sit down with Seiji Eicher, Distributed LLM Inference Engineer at Anyscale to unpack how large MoE models are actually served in practice, what it looks like to work deep in the LLM inference stack & more 🕐 Timestamps: 𝟬𝟬:𝟬𝟬 Why MoE comes up in modern AI infra 𝟬𝟬:𝟮𝟯 What Mixture of Experts actually changes inside a model 𝟬𝟭:𝟮𝟰 Why expert parallelism exists (and what becomes inefficient without it) 𝟬𝟮:𝟱𝟳 When MoE helps in practice 𝟬𝟰:𝟮𝟯 How Ray Serve fits into real inference stacks 𝟬𝟳:𝟬𝟳 What a Distributed LLM Inference Engineer works on day to day 𝟭𝟭:𝟭𝟵 Where the space is headed and how to learn more If you want to go deeper, Seiji gave a Ray Summit talk that dives into production-grade serving of MoE models, including KV-cache optimization, parallelism strategies, and multi-node orchestration with Ray Serve and vLLM. Check out the talk linked in the comments 👇

60

3 Comments

Google Research Director Denny Zhou, who founded the LLM Reasoning Team at DeepMind, recently gave a great talk at Stanford’s CS25 class: Large Language Model Reasoning. This was one of the most intuitive talks about reasoning model training and application. He outlined new directions for training models to handle questions where answers aren’t easily verifiable. My team is currently working on an enterprise LLM project focused on using reasoning to extract relevant context and generate responses for unverifiable cases. It’s a tough but promising area, and it’s an interesting time to be building. [Link to Denny Zhou's talk: https://lnkd.in/gdJ6Mzi8]

1,026

8 Comments

Super excited to launch a new AI course! 🚀 Fine-Tuning & Reinforcement Learning for LLMs: Intro to Post-Training.  🎓 Learn for free: https://lnkd.in/gxyqfxsF Created by the teams at AMD 🤝 Andrew Ng’s DeepLearning.AI. What you’ll learn: ▫️ How post-training works in frontier labs ▫️ How to use fine-tuning and RL to solve real business problems - hands-on! ▫️ How it all works: the intuition, the math, and how it enables reasoning and agentic behavior All powered by AMD GPUs – compute designed to support you, whether you’re just tinkering or scaling your next AI breakthrough. The future of AI belongs to everyone who builds it. Our mission is to make things open, so we can bring that future to life together: open knowledge, open software, open compute access. Let us know what else you want to learn! ❤️

815

34 Comments

Excited to share our NeurIPS 2025 Tutorial on How to Build Agents to Generate Kernels for Faster LLMs (and Other Models!) A collaboration across institutions: AMD, Stanford, Google DeepMind, Arm, Nvidia, Meta, Modular AI, UC Irvine, and ML Commons. - If you're an AI researcher, check out the parts of AI compute (e.g. GPU) that affect your inference & training jobs -- from why you should do batch sizes in certain multiples of 2 to how your trillions of matrix multiplications make their way through different types of memory on a chip - Also chat with us about the trend of self-improving LLMs that can write code to make them even faster. Agents -> SFT data -> RL with profiling tools for strong reward signals - Come build with us! It's an exciting future where LLMs can make the next generation of AI research (RL, even more sparsity, hybrid models with SSMs) super fast on any new hardware generation -- on-the-fly in your IDE Sina Rafati, Hao Li, Azalia Mirhoseini, Anna Goldie, Laurence Moroney, Vartika Singh, Mark Saroufim, Chris Lattner, Sitao Huang, David Kanter, Simon Guo, Kesavan Ramakrishnan, Vincent Ouyang, Tim Gianitsos, Nithyashree Manohar, Sharon Zhou, with acknowledgements to the AMD GEAK team Zicheng Liu, Dong Li, Ziqiong Liu, Pratik Prabhanjan Brahma and the AMD ROCm Kernel team ZHAOYI LI and the AMD Omni team Muhammad Awad, Keith Lowery, Cole Ramos and kernel engineers John Tyler, Muhammad Osama and of course the one and only Emad Barsoum. When they're out, will drop... * arxiv paper link * Github repo link

1,299

39 Comments

I’m excited to attend NeurIPS workshop sessions at San Diego and present two papers authored by the applied research team: 1. GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplace. Mikołaj Sacha et al. - NeurIPS Workshop on Unifying Representations in Neural Models (UniReps) 2. Towards Real-World Evaluation of Agentic Work in Freelance Marketplaces. Mattie Terzolo et al. - NeurIPS Workshop on LLM Evaluation Over the next couple of days, I’ll be sharing insights and some behind-the-scenes perspectives on how we think about these models, how they’re shaping Upwork today, and the opportunities we see ahead. For attendees already onsite at the main conference, please stop by the Upwork booth #722 to learn more about the efforts the ML & AI team is driving - from research to real product impact. Feels great to be again publishing at NeurIPS after a hiatus and looking forward to great conversations and inspiration ahead!

49

2 Comments

We just released new work on Quantization-Aware Distillation (QAD) for NVFP4, aimed at making 4-bit inference practical without giving up model quality. NVFP4 is NVIDIA’s 4-bit floating point format built for high throughput and low memory on Blackwell GPUs. The usual tradeoff with 4-bit is accuracy. QAD addresses this by distilling a low-precision student directly from a high-precision teacher using the full output distribution, not just labels. With QAD, NVFP4 models retain up to 99.4% of BF16 accuracy across multiple Nemotron checkpoints, while delivering much higher efficiency. On Blackwell, NVFP4 enables up to 4x higher math throughput than BF16 and 1.7X lower memory use than FP8. That directly translates into better inference cost, scale, and deployment flexibility. If you care about deploying low-precision models without sacrificing quality, this is worth a read. Blog: https://lnkd.in/gnwXCpb3 Paper: https://lnkd.in/gyGPpNrZ HF checkpoints: https://lnkd.in/gyGPpNrZ

543

15 Comments

🔥 Exciting News in Semantic Caching Technology! 🔥 Redis's AI Research team has just released langcache-embed-v1 - a powerful new embedding model specifically designed for semantic caching of LLMs! Check it out on Hugging Face: https://lnkd.in/gGukYc3k Fine-tuned from gte-modernbert-base this model, maps sentences and paragraphs to 768-dimensional dense vector spaces with impressive performance metrics: 90% cosine accuracy 87% F1 score 92% average precision 💡 Why this matters: Semantic caching dramatically reduces redundant LLM calls, cutting both latency and costs in production AI systems. This purpose-built model optimizes specifically for caching scenarios rather than general embeddings. 📑 Technical highlights: 8192 token sequence length Built on ModernBERT architecture Optimized with Online Contrastive Loss Simple integration with sentence-transformers library Learn more about "Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data" in their recent paper (arxiv:2504.02268). #Redis #AI #MachineLearning #SemanticCaching #LLM #Embeddings #AIPerformance #RedisEnterprise

106

1 Comment

I'm happy to share that we've released Instella-Math: a reasoning-focused language model trained with long CoT reinforcement learning on AMD GPUs. It is fully open – its base model (Instella-3B-Instruct), reasoning SFT, and RL training are all fully open. In addition to commonly used math and science reasoning benchmarks, we also evaluate on a new benchmark, TTT-Bench, targeting strategic, spatial, and logical reasoning. Our model performs surprisingly well on TTT-Bench without any exposure to similar-style data during any stage of training. #AMD #LLM #ML #AI #HW #MI300X https://lnkd.in/gnkFp7uQ

206

11 Comments

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

Excited to share that the Hammerspace team is in Santa Clara this week for an intensive AI Advisor training session at NVIDIA! We’re diving deep into the latest advancements in GPU-accelerated infrastructure, AI data pipelines, and how to best architect global data environments that connect generative, agentic and physical AI. As the demand for AI explodes, one thing is crystal clear: performance is no longer just about compute — it’s about making exabytes of unstructured data instantly available to GPU clusters anywhere in the world, without friction or copying. That’s the Hammerspace mission, and this training is arming our team with even more expertise to help customers build faster, simpler, and more efficient AI factories. Grateful to the incredible NVIDIA team for the world-class instruction and to our crew for leveling up. The future of AI is being built on data, and we’re proud to be part of making it unstoppable. #AI #GenerativeAI #GPU #DataForAI #AIanywhere #Hammerspace #NVIDIA 🚀

94

5 Comments

We built ProfBench to raise the bar for LLMs - literally. At NVIDIA, we worked with domain experts to create a benchmark that goes far beyond trivia and short answers. ProfBench tests LLMs on complex, multi-step tasks that demand the kind of reasoning, synthesis, and clarity you'd expect from a PhD physicist or MBA consultant. 🌎 This isn’t just a dataset drop. It’s a global collaboration: 38 professionals across 8 countries contributed over 7,000 expert-written rubrics across finance MBA 💵, consulting MBA 📊, chemistry PhD 🧪and physics PhD 🚀. 🧗Every prompt and grading rubric was handcrafted, requiring tens of hours of dedicated and focussed work. Now fully supported in the NeMo Evaluator SDK, ProfBench enables reproducible, rubric-based evaluations and side-by-side model comparisons. 🔗 ProfBench on Hugging Face https://lnkd.in/g2mCMcnc 🔗 NeMo Evaluator SDK https://lnkd.in/gF6SQCwt I’m so proud of the team that made this happen. Let’s keep pushing what AI can do. #ProfBench #LLM #AIevaluation #NeMo #NVIDIA #OpenSourceAI #AIresearch #AgenticAI #GenerativeAI #BuiltByExperts #GTCDC Work done with Jaehun Jung Ximing Lu Shizhe Diao Ellie Evans Jiaqi Zeng Pavlo Molchanov Yejin Choi Jan Kautz Yi Dong Collaborators: Vivienne Zhang Isabel Hulseman, MBA Seph Mard Pablo Ribalta, PhD Grzegorz Chlebus Wojciech Prazuch Ankit J. Patel

205

3 Comments