Yuting Chen

We built ProfBench to raise the bar for LLMs - literally. At NVIDIA, we worked with domain experts to create a benchmark that goes far beyond trivia and short answers. ProfBench tests LLMs on complex, multi-step tasks that demand the kind of reasoning, synthesis, and clarity you'd expect from a PhD physicist or MBA consultant. 🌎 This isn’t just a dataset drop. It’s a global collaboration: 38 professionals across 8 countries contributed over 7,000 expert-written rubrics across finance MBA 💵, consulting MBA 📊, chemistry PhD 🧪and physics PhD 🚀. 🧗Every prompt and grading rubric was handcrafted, requiring tens of hours of dedicated and focussed work. Now fully supported in the NeMo Evaluator SDK, ProfBench enables reproducible, rubric-based evaluations and side-by-side model comparisons. 🔗 ProfBench on Hugging Face https://lnkd.in/g2mCMcnc 🔗 NeMo Evaluator SDK https://lnkd.in/gF6SQCwt I’m so proud of the team that made this happen. Let’s keep pushing what AI can do. #ProfBench #LLM #AIevaluation #NeMo #NVIDIA #OpenSourceAI #AIresearch #AgenticAI #GenerativeAI #BuiltByExperts #GTCDC Work done with Jaehun Jung Ximing Lu Shizhe Diao Ellie Evans Jiaqi Zeng Pavlo Molchanov Yejin Choi Jan Kautz Yi Dong Collaborators: Vivienne Zhang Isabel Hulseman, MBA Seph Mard Pablo Ribalta, PhD Grzegorz Chlebus Wojciech Prazuch Ankit J. Patel

200

3 Comments

New work on optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a new perspective on spending test-time compute: Paper + Code: https://lnkd.in/eUfzZSaz This viewpoint allows us to interpret the long output stream from an LLM as a series of test-time episodes, leading us to use a notion of cumulative regret over output tokens as a metric for measuring test-time compute efficiency. Similar to how RL algorithms can balance exploration and exploitation over training, minimizing cumulative regret can also provide the best tradeoff between exploration and exploitation in the token stream. Using these insights, we develop Meta Reinforcement Fine-Tuning, or MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning compared to outcome-reward RL. Joint work with Yuxiao Qu, Matthew Yang, Amrith Setlur, Lewis Tunstall, Edward Beeching, Aviral Kumar

394

5 Comments

Today marks another step toward making Reinforcement Learning (RL) faster, simpler, and scalable for builders harnessing the Essential Cloud for AI. Our support on the development of torchforge - a new PyTorch-native framework - is helping researchers take large-scale RL workloads from experimentation to production. In partnership with Meta and Stanford University, torchforge was validated on CoreWeave’s Cloud Platform proving stable, efficient, and highly performant – ready to power the next generation of RL workloads. torchforge is now supported on SUNK. More in our recent blog: https://hubs.la/Q03PLJC10

136

Excited to share our new paper on Step-Wise Reinforcement Learning (SWiRL), which uses reinforcement learning and synthetic trajectories to improve multi-step reasoning and tool use! In our experiments on open-source Gemma 2 models, this post-training improved relative accuracy by 21.5%, 12.3%, 14.8%, 11.1%, and 15.3% on GSM8K, HotPotQA, CofCA, MuSiQue, and BeerQA, respectively. Interestingly, we observed cross-domain generalization: training on multi-hop question-answering improved performance on multi-step mathematical reasoning and vice versa! So, how does SWiRL work? This method has two phases: (1) Synthetic Data Generation: Iteratively prompt a model to generate multi-step reasoning and tool use trajectories, and then filter based on the “reasonableness” of each step individually. (2) Step-Wise Reinforcement Learning: Decompose synthetic trajectories into contextualized prompts for each action by the model, and then reward the model based on its proposed next action in the context of all previous actions. VentureBeat wrote a nice article in which we share more thoughts: https://lnkd.in/g3KZkW8A For more details, please check out our paper: https://lnkd.in/gBYrncer Many thanks to my amazing coauthors Azalia Mirhoseini, Christopher Manning, Irene Cai, and Hao Zhou!

128

1 Comment

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Let's learn more! In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point—MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable—without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. I created it while taking inspiration from Femke Plantinga's post on this topic. If you want to understand this better with code, I have linked my article in the comments. ____ If you want to learn AI/ML engineering, I have put together a free PDF (530+ pages) with 150+ core DS/ML lessons. Get here: https://lnkd.in/gi6xKmDc ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

1,383

48 Comments

The PyTorch Foundation is excited to welcome vLLM as a PyTorch Foundation-hosted project. vLLM is a high-throughput, memory-efficient inference and serving engine designed for LLMs. vLLM has always had a strong connection with the PyTorch project. It is deeply integrated into PyTorch, leveraging it as a unified interface to support a wide array of hardware backends. These include NVIDIA GPUs, AMD GPUs, Google Cloud TPUs, Intel Corporation GPUs, Intel CPUs, Intel Gaudi HPUs, and Amazon Web Services (AWS) Neuron, among others. This tight coupling with PyTorch ensures seamless compatibility and performance optimization across diverse hardware platforms. 📢 Read the full announcement: https://hubs.la/Q03lmH_F0 #vLLM #PyTorchFoundation #PyTorch #OpenSourceAI

754

7 Comments

📑 New Survey: The Landscape of Agentic Reinforcement Learning for LLMs A comprehensive survey has just been released, synthesizing 500+ recent works on how reinforcement learning is transforming large language models from passive text generators into agentic decision-makers. 🔑 Key highlights from the paper: - Formalizes the shift from LLM-RL (single-step MDPs) to Agentic RL (long-horizon POMDPs) - Proposes a twofold taxonomy: By capabilities: planning, reasoning, tool use, memory, self-improvement, perception By applications: code agents, math agents, GUI agents, vision/embodied agents, multi-agent systems, and more - Consolidates open-source environments, frameworks, and benchmarks into a practical compendium for researchers - Discusses open challenges: trustworthiness, scaling agentic training, and scaling environments

113

4 Comments

🚀𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗟𝗠 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 We’re excited to share our latest work on improving inference-time efficiency for LLMs through KV cache quantization—a critical step toward making long-context reasoning more scalable and memory-efficient. 🧠𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹𝘀 & 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲-𝘁𝗶𝗺𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 Modern reasoning models often require long responses to “think” through problems before arriving at a final answer. Inference-time scaling methods make this even more compute-intensive. While these approaches improve model performance, they incur higher latency and demand more GPU memory. 💡𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 The KV cache stores the intermediate representations from previous tokens to accelerate autoregressive decoding. Think of it as the model’s short-term memory—just as humans recall previous parts of a conversation to respond quickly, the KV cache helps models maintain and build on prior context. For long sequences, the KV cache can consume more GPU memory than the model weights. During inference, LLM decoding becomes memory-bound, with most of the time spent on data transfer rather than computation. This has led to active research on KV cache quantization but quantization errors can accumulate as more tokens are generated, causing later tokens to deviate from expected outputs. ✨𝗪𝗵𝗮𝘁’𝘀 𝗻𝗲𝘄 𝗶𝗻 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? We introduce 𝚂̲𝚀̲𝚞̲𝚊̲𝚝̲ (Subspace-orthogonal KV cache quantization)—a new method that significantly reduces memory overhead and latency while maintaining model accuracy. SQuat constructs a subspace that captures critical task-relevant information, then enforces quantization errors to lie orthogonal to this subspace, minimizing their effect on the output of the attention mechanism. 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: ✅Training-free: No fine-tuning or calibration data needed ✅On-the-fly: Runs during inference without modifying the model ✅Theory-grounded: Built on a theoretical foundation we developed ⚡𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁: • Reduces GPU peak memory by 2.17× to 2.82× • Improves throughput by 2.45× to 3.60× • outperform existing KV cache quantization methods on benchmark tasks 📄paper: https://lnkd.in/emKhAVZu 💻 code: https://lnkd.in/e8TJ7N3R 👏 Joint work with my amazing co-authors Ligong Han, Kai Xu, Akash Srivastava, at the Red Hat AI innovation team (https://lnkd.in/exd6QDbk) Chris Wright, Ruchir Puri, Steven Huels, Joe Fernandes, Tushar Katarki, Ritika Gunnar, Jason McGee, Máirín Duffy, Luke Inglis, Mark Kurtz, Nick Hill, Tyler Michael Smith, vLLM 

206

1 Comment

I first got started with Reinforcement Learning in 2016-2017, when it still lived mostly in research circles after the Atari results from DeepMind using RL. Fast forward to today, RL is having a moment again, largely because it has become a prominent way to make LLMs learn better. I’m glad to see the surge in interest, so I put together a quick handout on the basics. I’ll follow up with more posts to break down key ideas and how they’re used in practice. Here’s a crisp primer to get you started (in the handout below) If you’re curious to learn, grab the handout and stay tuned. I’ll share follow-ups on PPO, reward modeling, DPO vs RLHF, and practical pitfalls to avoid. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

261

37 Comments

An exciting new course: Fine-tuning and Reinforcement Learning for LLMs: Intro to Post-training, taught by Sharon Zhou, PhD, VP of AI at AMD. Available now at DeepLearning.AI. Post-training is the key technique used by frontier labs to turn a base LLM--a model trained on massive unlabeled text to predict the next word/token--into a helpful, reliable assistant that can follow instructions. I've also seen many applications where post-training is what turns a demo application that works only 80% of the time into a reliable system that consistently performs. This course will teach you the most important post-training techniques! In this 5 module course, Sharon walks you through the complete post-training pipeline: supervised fine-tuning, reward modeling, RLHF, and techniques like PPO and GRPO. You'll also learn to use LoRA for efficient training, and to design evals that catch problems before and after deployment. Skills you'll gain: - Apply supervised fine-tuning and reinforcement learning (RLHF, PPO, GRPO) to align models to desired behaviors - Use LoRA for efficient fine-tuning without retraining entire models - Prepare datasets and generate synthetic data for post-training - Understand how to operate LLM production pipelines, with go/no-go decision points and feedback loops These advanced methods aren’t limited to frontier AI labs anymore, and you can now use them in your own applications. Learn here: https://lnkd.in/gn9UAunn

3,849

95 Comments

𝐏𝐚𝐫𝐞𝐭𝐨𝐐: 𝐒𝐜𝐚𝐥𝐢��𝐠 𝐋𝐚𝐰𝐬 𝐢𝐧 𝐄𝐱𝐭𝐫𝐞𝐦𝐞𝐥𝐲 𝐋𝐨𝐰-𝐛𝐢𝐭 𝐋𝐋𝐌 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 Quantization of LLMs has emerged as a pivotal research area, driven by the imperative to reduce computational and memory demands while preserving model performance. However, the optimal bit-width for achieving the best tradeoff between quantized model size and accuracy has been a subject of ongoing debate. To address these multifaceted challenges, we propose 𝐏𝐚𝐫𝐞𝐭𝐨𝐐, an advanced quantization framework that achieves SOTA performance across all bit-width levels. Our empirical analysis indicates that quantization at 1.58-bit, 2-bit, and 3-bit offers a superior trade-off between accuracy and effective quantized model size compared to 4-bit, highlighting their potential for optimized model deployment. Paper: https://lnkd.in/ga7rP7Qb

206

2 Comments