Gary Li, Ph.D.

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better��TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Let's learn more! In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point—MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable—without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. I created it while taking inspiration from Femke Plantinga's post on this topic. If you want to understand this better with code, I have linked my article in the comments. ____ If you want to learn AI/ML engineering, I have put together a free PDF (530+ pages) with 150+ core DS/ML lessons. Get here: https://lnkd.in/gi6xKmDc ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

1,384

48 Comments

I’ve noticed that many GenAI application projects put in automated evaluations (evals) of the system’s output probably later — and rely on humans to manually examine and judge outputs longer — than they should. This is because building evals is viewed as a massive investment (say, creating 100 or 1,000 examples, and designing and validating metrics) and there’s never a convenient moment to put in that up-front cost. Instead, I encourage teams to think of building evals as an iterative process. It’s okay to start with a quick-and-dirty implementation (say, 5 examples with unoptimized metrics) and then iterate and improve over time. This allows you to gradually shift the burden of evaluations away from humans and toward automated evals. I wrote previously in The Batch about the importance and difficulty of creating evals. Say you’re building a customer-service chatbot that responds to users in free text. There’s no single right answer, so many teams end up having humans pore over dozens of example outputs with every update to judge if it improved the system. While techniques like LLM-as-judge are helpful, the details of getting this to work well (such as what prompt to use, what context to give the judge, and so on) are finicky to get right. All this contributes to the impression that building evals requires a large up-front investment, and thus on any given day, a team can make more progress by relying on human judges than figuring out how to build automated evals. I encourage you to approach building evals differently. It’s okay to build quick evals that are only partial, incomplete, and noisy measures of the system’s performance, and to iteratively improve them. They can be a complement to, rather than replacement for, manual evaluations. Over time, you can gradually tune the evaluation methodology to close the gap between the evals’ output and human judgments. For example: - It’s okay to start with very few examples in the eval set, say 5, and gradually add to them over time — or subtract them if you find that some examples are too easy or too hard, and not useful for distinguishing between the performance of different versions of your system. - It’s okay to start with evals that measure only a subset of the dimensions of performance you care about, or measure narrow cues that you believe are correlated with, but don’t fully capture, system performance. For example if, at a certain moment in the conversation, your customer-support agent is supposed to (i) call an API to issue a refund and (ii) generate an appropriate message to the user, you might start off measuring only whether or not it calls the API correctly and not worry about the message. Or if, at a certain moment, your chatbot should recommend a specific product, a basic eval could measure whether or not the chatbot mentions that product without worrying about what it says about it. [Truncated due to length limit. Full text: https://lnkd.in/gygj3y7w ]

4,071

178 Comments

[AI paper] GRPO has a hidden flaw in Multi-Reward settings. It’s time to decouple your normalization. 📉 As AI Architects, we are moving from single-objective RL (just "get the answer right") to multi-objective RL (accuracy + format + length + safety). We typically sum these rewards and throw them into GRPO. A new paper from NVIDIA, "GDPO: Group reward-Decoupled Normalization Policy Optimization," demonstrates that this naive summation causes Reward Signal Collapse. The Architectural Failure Mode: If you sum disparate rewards (e.g., a binary Format reward and a scalar Accuracy reward) and then normalize, different raw reward combinations can map to identical advantage values. Example: A rollout with rewards (0, 2) might yield the same normalized advantage as (0, 1) due to group statistics, effectively deleting the gradient signal for the second objective . The Solution: GDPO The fix is architectural: Decoupled Normalization. Instead of normalizing the sum, GDPO normalizes each reward objective independently within the group before aggregation. Impact: This restores the resolution of the training signal. On Tool Calling and Math Reasoning tasks (DeepSeek-R1/Qwen), GDPO converges where GRPO fails or plateaus. If you are training agents that must balance strict formatting (JSON) with reasoning quality, this is the correct objective function to use. 👇 Link to the paper in the comments. #AIArchitecture #RLHF #NVIDIA #GRPO #DeepLearning #LLM #Alignment #Research

21

1 Comment

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here: https://lnkd.in/gdjae8AX

4,672

190 Comments

Model training on TPUs just got way easier! A completely reimagined vLLM TPU for LLM inference 👉 for PyTorch and JAX developers, this means more flexibility to run PyTorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX. read: https://lnkd.in/gW7476Hs #GoogleCloud #TPUs #LLMs

14

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

We are moving from AI that thinks to agents that act. A massive new survey paper involving Google DeepMind and UIUC defines the next frontier of intelligence: agentic reasoning. For the last two years, the industry has focused on 'thought' (reasoning in closed-world settings like maths or code). but the real world is open-ended, dynamic, and messy. the breakthrough isn't just a smarter model: it is a paradigm shift in how AI interacts with reality. The researchers break this down into a systematic roadmap that every governance leader needs to understand: - foundational reasoning: the core ability to plan, use tools, and learn from experience. - evolutionary reasoning: how agents adapt and refine their strategies over time. - collaborative reasoning: the most critical layer for governance. how multiple agents coordinate to solve problems without human intervention. Why is this the most important paper for AI governance right now? Because agentic reasoning bridges the gap between 'thought' and 'action'. when an AI can plan its own steps and use tools to execute them, the traditional 'human-in-the-loop' model becomes a bottleneck. We are moving away from treating AI as a search engine and toward treating it as an autonomous workforce. This brings us back to the core challenge: intelligent delegation. As these agents gain the ability to collaborate and act in the real world, our governance cannot just be about 'safety filters', it must be about building the protocols for authority, responsibility, and trust in a multi-agent ecosystem. The era of the 'static chatbot' is ending and the era of the 'agentic colleague' has begun. #AI #AgenticAI #DeepMind #AIGovernance #FutureOfWork #TechLeadership

89

20 Comments

One of the persistent challenges in using large language models (LLMs) is getting them to follow instructions reliably — especially when the instructions are subtle or domain-specific. DeepMind’s latest research introduces Symbol Tuning, a simple yet powerful fine-tuning method that significantly improves an LLM’s ability to follow symbolic prompts (e.g., bullet points, XML, Markdown, or code-like instructions) in zero-shot and few-shot settings. https://lnkd.in/gzKDdHQ2 Why this matters: 🔹 Improves instruction following in GPT-class models 🔹 Works with tiny amounts of data (just 100K tokens!) 🔹 Boosts performance in math, code, and reasoning-heavy tasks 🔹 Enhances models' ability to generalize across symbolic formats This has massive implications for building enterprise agents, RAG pipelines, and developer copilots that need high-precision, structured interaction with users or data. A great reminder: sometimes, small, well-targeted innovations create massive gains. #LLM #InContextLearning #SymbolTuning #PromptEngineering #DeepMind #GenAI #AIResearch #InstructionFollowing #EnterpriseAI #DeveloperTools

199

31 Comments

As someone deeply invested in agentic systems and reinforcement learning, I find the Chain-of-Agents (CoA) approach both refreshing and practical, it takes the chaos of multi-agent orchestration and distills it into something elegant: a single model with native agent-like behavior. Here’s what I see: Traditional multi-agent setups have been great for modularity and reasoning, but they come with a high tax, too much inter-agent communication, redundant memory syncing, and unnecessary tool calls. What CoA proposes is radical in its simplicity: distill the behaviors of successful multi-agent runs into one model, then train that model to act as a team, internally. They start with multi-agent distillation: successful traces from existing orchestrators like OAgents are transformed into CoA-style traces, complete with planning steps, tool usage, reflections, and execution paths. But they don’t stop at replay, they filter hard cases and focus on high-quality trajectories that demonstrate useful agent behaviors like tool efficiency and coherent planning. This becomes the foundation of their SFT phase. Once that base is established, Agentic RL kicks in. The model is tasked with solving complex reasoning problems and importantly, ones where tools make or break the outcome (web QA, coding, math). Rewarding is done via exact-match or LLM-as-Judge strategies, making it possible to scale reward signals across noisy tasks. This setup tunes the model to reason reflectively, use tools judiciously, and respond with stability under diverse challenges. The kicker: by collapsing the roles of multiple agents into a single coherent model, CoA delivers an 84.6% reduction in inference cost. It’s a major unlock for production-grade agent systems where latency, memory, and tool limits matter. What excites me most is how CoA generalizes the spirit of ReAct/TIR, but upgrades it, dynamically invoking roles, using tools only when needed, and preserving a unified memory state that avoids the “who’s speaking now?” confusion. To me, Chain-of-Agents is a promising new frontier: a single-model, multi-mind architecture that captures the power of coordination while optimizing for real-world deployment constraints. And that’s where agentic AI needs to go. https://lnkd.in/exTxvsDD

63

2 Comments

Very excited to present with Haocheng Bian on how we leverage Ray and PyTorch to build two high-performance ML frameworks at Apple: • Aurora — focuses on deep learning and lightweight LLM training, leveraging FSDP and DeepSpeed. • Aether — focuses on massive-scale pre-training and RL, supporting models with hundreds of billions of parameters through PyTorch’s multi-dimensional parallelism. With support for distributed training, advanced checkpointing, mixed-precision training, kernel optimizations, deterministic DataLoader and worker checkpoint/recovery, these frameworks enable reliable and scalable LLM training across thousands of GPUs. Most importantly, the system maximizes flexibility: all framework interfaces are abstracted into a clean, notebook-style workflow—removing infrastructure complexity while giving researchers full freedom to build on new capabilities and iterate rapidly Talk link: https://lnkd.in/enqNv2Fu

178

1 Comment

Large Language Models (LLMs) excel at generating fluent text at a local level, but LLMs struggle to consistently maintain the structure and coherence of generated text over long documents and across different levels of detail. Techniques proposed in this paper introduce a system for enhancing text generation through a structured layered synthesis approach.

18

Andrew Ng is right: vertical AI is where the value is. Andrew Ng said something on last week's 20VC podcast that captures what we've learned building Diffusely: the real value in AI isn't in foundation models - it's in vertical applications that transform specific industries. As Ng put it: "Moats tend to be a function of the industry rather than a function of the technology." This isn't about having better models than OpenAI or Google. It's about industry-specific data, workflows and integrations that horizontal tools can't replicate. CarCutter integrates with DMS systems and understands dealership workflows. AutoRetouch handles brand consistency across thousands of SKUs. Both leverage our 200M image library of professional photography. Horizontal AI tools can't do this - not because their models aren't good enough, but because they're not built for it. Ng emphasizes that AI's value isn't saving 20% on one task. It's redesigning workflows for transformation. One automotive client reduced photography time from 28 to 5 minutes per vehicle. €600k saved annually. Fashion brands achieve 90% faster time-to-market. This is 10x gains - fundamental changes to operations, not incremental improvements. The current narrative is that foundation model companies will dominate. Ng's thesis - and our experience - suggests otherwise. Foundation models are infrastructure. Valuable, but not where economic value gets captured. The value is in vertical applications built on proprietary data that integrate deeply with industry workflows. Vertical AI requires understanding the industry as deeply as you understand AI. Professional services, not just software. Patience to integrate with legacy systems. Most AI companies optimize for horizontal scale because it's faster. Building vertical AI is slower, requires more expertise. 𝐓𝐡𝐚𝐭'𝐬 𝐰𝐡𝐲 𝐢𝐭'𝐬 𝐝𝐞𝐟𝐞𝐧𝐬𝐢𝐛𝐥𝐞. Three years building vertical AI convinced us Ng is right. The companies that win won't just have the best models. They'll have the deepest vertical expertise, specialized data, and strongest integration into industry workflows. #verticalai #ai #moat #entrepreneurship #enterprisesaas

35

1 Comment

Search relevance is the backbone of any RAG system. If the retrieval is bad, the answers are bad — no matter how good the LLM is. We just published a deep dive on how we approach this at Dropbox Dash, and I think the core idea is worth sharing because it applies well beyond our product. The challenge: Dash searches across millions (sometimes billions) of enterprise documents to find the handful that matter for a given query. A relevance model ranks those results, and that model needs training data — lots of it. Human labeling is the gold standard, but it's expensive, slow, and can't touch sensitive customer data. Pure LLM labeling is cheap and fast, but uncalibrated LLMs make systematic mistakes. Our approach: use humans to teach the LLM, then use the LLM to scale. A small set of human-labeled examples calibrates the LLM's relevance judgments. Once the LLM meets quality thresholds, it generates hundreds of thousands of labels that train the production ranking model. The result is a 100x amplification of human labeling effort. A few things that made this work in practice: We focused evaluation on the hardest cases — documents users clicked that the LLM rated low, and documents users skipped that the LLM rated high. That's where the learning happens. We gave the LLM tools to research context before judging. Enterprise search is full of internal acronyms and jargon (at Dropbox, "diet sprite" is a performance management tool, not a soft drink). Without context, even strong models get these wrong consistently. We used DSPy (Community) to programmatically optimize prompts against human judgments, which turned prompt tuning from an art into something measurable and repeatable. The pattern — small human reference set, calibrated LLM evaluation, scaled labeling, continuous monitoring — is generalizable to any domain where you need reliable judgments at scale. Full our post on the Dropbox tech blog: https://lnkd.in/gB_5TECm

68

Just getting started with LLMs? Don’t let the jargon slow you down. This 5-part LLM Glossary is your shortcut to mastering the fundamentals👇 1️⃣ Model Types • Foundation Model: Pretrained on large-scale, diverse datasets to learn general capabilities. • Instruction-Tuned Model: Further trained to follow user instructions precisely. • Multi-modal Model: Handles multiple input/output formats like text, images, and audio. • Reasoning Model: Optimized for logic, problem-solving, and step-by-step thinking. • Small Language Model (SLM): Lightweight models for fast, efficient, on-device tasks. 2️⃣ Training & Fine-Tuning • Pretraining: Initial learning phase using massive, diverse data sources. • RLHF (Reinforcement Learning with Human Feedback): Aligns model behavior with human preferences. • DPO (Direct Preference Optimization): Uses ranked preferences instead of rewards for training. • Synthetic Data: AI-generated data used to supplement real datasets. • Fine-Tuning: Targeted retraining on specific domains or tasks. • LoRA / QLoRA: Techniques for efficient low-resource fine-tuning. • Guardrails: Rules or filters applied to enforce safety, ethics, and compliance. 3️⃣ Prompt Engineering • System/User Prompts: Define model behavior and user context in a session. • Chain-of-Thought (CoT): Prompts that guide the model to reason step-by-step. • Few-Shot / Zero-Shot Learning: Demonstrating tasks with few or no examples. • Prompt Tuning: Optimizing prompts via training for specific outcomes. • Context Window: The total number of tokens the model can "remember" per interaction. 4️⃣ Inference & Generation • Temperature: Adjusts output randomness; lower values are more deterministic. • Max Tokens: Limits the number of tokens in a generated response. • Seed: Controls reproducibility of outputs. • Latency: Time taken by the model to respond. • Hallucination: When the model generates factually incorrect but plausible-sounding information. 5️⃣ Retrieval-Augmented Generation (RAG) • Retrieval: Fetching relevant data from external sources before generation. • Semantic Search: Search based on meaning, not just keywords. • Embeddings: Vector representations of text used for similarity matching. • Chunks: Dividing documents into smaller segments for better retrieval. • Vector Databases (VectorDBs): Stores embeddings for fast and accurate search. • Reranking: Rescoring retrieved documents to prioritize relevance. • Indexing: Structuring data for efficient retrieval during generation. 🌟 Follow the AIKaDoctor (Free AI & Data Science Resources) channel on WhatsApp: https://lnkd.in/dCTCEKKc 📌Follow Habib Shaikh For more such content.

232

36 Comments

Excited to share a recent paper: “Out-of-Context Abduction: LLMs Make Inferences About Procedural Data Leveraging Declarative Facts in Earlier Training Data” by Imran et al. This work investigates whether large language models can infer the most plausible explanations for observations by retrieving relevant facts from their pre-training corpus—what the authors term out-of-context abduction. Key Contributions: • Defining Out-of-Context Abduction: Unlike in-context abduction, which relies on hypotheses presented alongside observations, this form tests whether LLMs can leverage declarative knowledge from earlier training to explain procedural examples. • Experiment 1: GPT-4o, after being declaratively fine-tuned on chatbot personas, correctly inferred the “Pangolin” persona (responds only in German) with an 84% posterior probability—significantly above its 49% prior—simply by observing German-language responses. • Experiment 2: Through iterative fine-tuning on responses with increasing proportions of vowel-starting words (the “Axolotl” persona), GPT-4o demonstrated accelerated trainability that aligns with abductive inference mechanisms. How They Achieved It: 1. Declarative Fine-Tuning: Hand-crafted Q&A pairs associating each persona’s name with abstract behavior descriptions were used to bias the model without exposing it to example dialogues. 2. In-Context Behavior Examples: Procedural demonstrations (e.g., German replies or vowel-word responses) were encoded separately from the declarative facts to prevent simple pattern matching. 3. Off-Policy Iterative Fine-Tuning: By binning generated responses according to characteristic scores and re-training on the highest-scoring subsets, the authors measured how prior knowledge boosts learning efficiency. Why It Matters: • Emergent Reasoning: Provides strong evidence that LLMs can perform multi-hop abductive reasoning—first hypothesizing a behavior description, then mapping it back to a class name—beyond mere surface pattern matching. • AI Safety & Alignment: Situational awareness via out-of-context abduction could enhance AI assistance but also poses risks like identity inference or evaluation gaming, underscoring the need for robust interpretability and monitoring. This research opens new avenues for exploring latent reasoning in LLMs and informs safer deployment strategies. #MachineLearning #LLM #AbductiveReasoning #AISafety #AIResearch #GenAI #AI

9

We’re excited to welcome Ray to the PyTorch Foundation 👋 Ray is an open source distributed computing framework for AI workloads, including data processing, model training and inference at scale. By contributing Ray to the PyTorch Foundation, Anyscale reinforces its commitment to open governance and long-term sustainability for Ray and open source AI.    “The PyTorch Foundation is committed to fostering an open, interoperable, and production-ready AI ecosystem. By bringing Ray under the PyTorch Foundation umbrella, alongside projects like vLLM and DeepSpeed, we are uniting the critical components needed to build next-generation AI systems. Ray’s inclusion strengthens our collective mission to support developers with the tools to efficiently train, serve, and deploy AI models at scale.” - Matt White GM of AI at The Linux Foundation Foundation and Executive Director of the PyTorch Foundation “With Ray, our goal is to make distributed computing as straightforward as writing Python code. Joining the PyTorch Foundation helps us stay true to that mission, ensuring Ray continues to be an open, community-driven backbone for developers and their organizations.” - Robert Nishihara, co-founder of Anyscale ➡️ Read the announcement: https://lnkd.in/e3e4mjds #Ray #PyTorchCon #PyTorch #AI #OpenSource #AICompute #DistributedComputing #AIWorkloads #DataProcessing #ModelTraining #Inference

668

5 Comments