Kumar Chellapilla

Introducing "Building with Llama 4." This short course is created with Meta, and taught by Amit Sangani, Director of Partner Engineering for Meta’s AI team. Meta’s new Llama 4 has added three new models and introduced a Mixture-of-Experts (MoE) architecture to its family of open-weight models, making them more efficient to serve. In this course, you’ll work with two of the three new models introduced in Llama 4. First is Maverick, a 400B parameter model, with 128 experts and 17B active parameters. Second is Scout, a 109B parameter model with 16 experts and 17B active parameters. Maverick and Scout support long context windows of up to a million tokens and 10M tokens, respectively. The latter is enough to support directly inputting even fairly large GitHub repos for analysis! In hands-on lessons, you’ll build apps using Llama 4’s new multimodal capabilities including reasoning across multiple images and image grounding, in which you can identify elements in images. You’ll also use the official Llama API, work with Llama 4’s long-context abilities, and learn about Llama’s newest open-source tools: its prompt optimization tool that automatically improves system prompts and synthetic data kit that generates high-quality datasets for fine-tuning. If you need an open model, Llama is a great option, and the Llama 4 family is an important part of any GenAI developer's toolkit. Through this course, you’ll learn to call Llama 4 via API, use its optimization tools, and build features that span text, images, and large context. Please sign up here: https://lnkd.in/gXKeipht

2,832

111 Comments

𝗪𝗵𝗮𝘁 𝗱𝗼𝗲𝘀 𝗶𝘁 𝗺𝗲𝗮𝗻 𝘁𝗼 𝘀𝗰𝗮𝗹𝗲 𝗰𝗼𝗻𝘀𝘂𝗺𝗲𝗿 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝘀 𝗶𝗻 𝗮𝗻 𝗔𝗜-𝗳𝗶𝗿𝘀𝘁 𝘄𝗼𝗿𝗹𝗱? Dived deep into this at the roundtable at Bits n Atoms, hosted by The Product Folks last night — and the conversations stuck with me. Scaling consumer products in an AI world demands thinking across multiple dimensions, and these are just some of the lenses through which our group explored it.  • 𝗣𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘀𝗮𝘁𝗶𝗼𝗻 — not just smart recommendations, but designing for one and actually impacting behaviour change  • 𝗧𝗿𝘂𝘀𝘁 — especially for this new generation of AI-native products, where UX becomes critical to adoption  • 𝗖𝗼𝘀𝘁-𝗲𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲𝗻𝗲𝘀𝘀 — using AI to boost efficiency and also designing AI systems that can scale without exploding infra or model costs  • 𝗠𝗼𝗻𝗲𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 — whether through subscriptions or new models that make sense for AI-first experiences  • 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻 & 𝗚𝗧𝗠 — this differentiation matters more than ever, in a world where building products is getting faster and cheaper everyday  • 𝗔𝗰𝗰𝗲𝘀𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 — across regions, languages, devices, and modalities. If it only works for the few, does it really scale? We’re in an interesting moment — where product thinking is expanding, not just because of what AI can do, but because of the questions it’s forcing us to ask. Loved the sharp, honest, and wide-ranging perspectives from everyone Shreyanshi C. Kushal Khandelwal Ashish Jain Madhav Bhartia Tanmay Saxena Pranav Agrawal Sachin Kamkar Shashank Raghavendra Nikhil Ankan Huge shout out to Abhay Jani Aditya Mohanty Suhas Motwani for driving these thoughtfully curated and highly impactful events!

92

6 Comments

In this week’s AI infra chat, we sit down with Seiji Eicher, Distributed LLM Inference Engineer at Anyscale to unpack how large MoE models are actually served in practice, what it looks like to work deep in the LLM inference stack & more 🕐 Timestamps: 𝟬𝟬:𝟬𝟬 Why MoE comes up in modern AI infra 𝟬𝟬:𝟮𝟯 What Mixture of Experts actually changes inside a model 𝟬𝟭:𝟮𝟰 Why expert parallelism exists (and what becomes inefficient without it) 𝟬𝟮:𝟱𝟳 When MoE helps in practice 𝟬𝟰:𝟮𝟯 How Ray Serve fits into real inference stacks 𝟬𝟳:𝟬𝟳 What a Distributed LLM Inference Engineer works on day to day 𝟭𝟭:𝟭𝟵 Where the space is headed and how to learn more If you want to go deeper, Seiji gave a Ray Summit talk that dives into production-grade serving of MoE models, including KV-cache optimization, parallelism strategies, and multi-node orchestration with Ray Serve and vLLM. Check out the talk linked in the comments 👇

60

3 Comments

Thanks to the resurgence of RL, LLMs are finally able to reliably coordinate tools and reasoning to do high-precision retrieval. Companies like Happenstance, Clado, and Mintlify have already shifted to agentic search, and it's only a matter of time until anything less feels broken to users. Link to full blog in comments.

20

2 Comments

Is AI any good at evaluating AI? Is it turtles all the way down? We benchmarked evaluation models like LLM-as-a-judge, HHEM, Prometheus across 6 RAG applications. Evaluation models work surprisingly well in practice. Hoping to see more of these real-time reference-free evaluations to give end users more confidence in the outputs of AI applications.

51

5 Comments

Back in 2023, TGI has initiated a movement where optimized inference engines rely on the model architectures from transformers as the source of modeling truth. This approach is now adopted by downstream inference engines: vLLM, SGLang, as well as local engines with inter-compatibility such as llama.cpp or MLX. Given we at Hugging Face contribute to those community runtimes, we decided to switch text-generation-inference itself to maintenance mode. Thanks for all the tokens 🔥 Hat/tip Lysandre Debut

480

32 Comments

🌐 Had a fascinating conversation with Sabeer Bhatia (Hotmail co-founder) at TiEcon about the future of computing—particularly quantum. The consensus? 👉 Quantum chips won’t replace digital chips. They’ll augment them—just like GPUs did for AI. We discussed emerging quantum modalities: Superconducting (IBM, Google) Trapped Ions (IonQ, Quantinuum) Photonics (Xanadu, PsiQuantum) Neutral atoms (ColdQuanta) Topological qubits (Microsoft) Some great insights from leaders in the field: 🧠 Chetan Nayak (Microsoft): "Most quantum systems today are like analog radios—fragile and noisy. With topological qubits, we’re building something closer to digital transistors: stable, scalable, and resilient." 🧠 Jay Gambetta (IBM): "Quantum won’t replace classical—it’s about expanding the computational toolbox. The future is hybrid: CPUs, GPUs, and QPUs solving what no one system can." 🚛 Arvind Ratnam (QCNTRL): "Quantum chips are already solving problems where GPS fails—underground, underwater, or in jammed environments. That’s game-changing for logistics, defense, and autonomy." 🔬 Use cases gaining traction: Drug discovery Logistics optimization Post-quantum encryption Quantum-enhanced AI It’s clear: Quantum computing is becoming a critical co-processor layer—not a replacement. The next decade of computing will be hybrid, intelligent, and cross-disciplinary. #QuantumComputing #AI #FutureOfTech #TiEcon #QuantumChips #Microsoft #IBM #QCNTRL #DeepTech #Innovation #HybridComputing

32

1 Comment

For the past few years, AI headlines have centered on training - bigger models, larger clusters, record-setting compute. But a structural shift underway is toward inference. As AI moves from experimentation to deployment at scale, inference will define responsiveness, energy efficiency, cost economics and real-world utility. This is no longer just a scaling problem. It’s an architectural one. The next chapter of AI will be written by those who can deliver intelligence efficiently, reliably, and sustainably at global scale.

168

22 Comments

Barbarians at the Gate (https://lnkd.in/gkEBX79s) is a very interesting new paper from some great folks at Berkeley, with some exciting results showing the potential for AI in systems research. But I think the authors aren't quite asking the hardest problem about where this takes systems as a field. I wrote a new blog post about it: https://lnkd.in/gxS3s4v4 The core point of the paper is that AI can be effective at automatically driving the 'number go up' part of systems research. Given an evaluator, AI can automate progress on implementation. I believe this is 100% correct. The paper also talks about how more focus will have to go into developing evaluators. Again, I 100% agree, but think they're understating their case. Building evaluators (basically tests) is, I believe, going to become the focus of systems and software over the next decade. Which leaves researchers with the hard problem of choosing which problems to go after. That's never been easy in academic systems, and is only going to become more of a challenge. But the best systems research is driven by choosing problems well! This vision for AI could lead to lots more of the least interesting systems research. The "number go up" kind. Which could make it harder than ever to identify the truly innovative and visionary work. We're already seeing that elsewhere in science. So, mostly, I think the paper is right, but not yet asking the biggest question: what should systems research be?

276

12 Comments

In a new blog by Smallest AI, we talk about issues in large fused kernels, particularly problems with: 1. Dynamic Input Shapes 2. Synchronization and Deadlocks 3. L1 cache latency compounding 4. Cross SM coordination 5. Warp context bottlenecks Further, we segway into what would it take to build a sub 50ms(and lower) speech model. Link and author name in comments, let us know what you think!

134

7 Comments