Manchen Wang

We built ProfBench to raise the bar for LLMs - literally. At NVIDIA, we worked with domain experts to create a benchmark that goes far beyond trivia and short answers. ProfBench tests LLMs on complex, multi-step tasks that demand the kind of reasoning, synthesis, and clarity you'd expect from a PhD physicist or MBA consultant. 🌎 This isn’t just a dataset drop. It’s a global collaboration: 38 professionals across 8 countries contributed over 7,000 expert-written rubrics across finance MBA 💵, consulting MBA 📊, chemistry PhD 🧪and physics PhD 🚀. 🧗Every prompt and grading rubric was handcrafted, requiring tens of hours of dedicated and focussed work. Now fully supported in the NeMo Evaluator SDK, ProfBench enables reproducible, rubric-based evaluations and side-by-side model comparisons. 🔗 ProfBench on Hugging Face https://lnkd.in/g2mCMcnc 🔗 NeMo Evaluator SDK https://lnkd.in/gF6SQCwt I’m so proud of the team that made this happen. Let’s keep pushing what AI can do. #ProfBench #LLM #AIevaluation #NeMo #NVIDIA #OpenSourceAI #AIresearch #AgenticAI #GenerativeAI #BuiltByExperts #GTCDC Work done with Jaehun Jung Ximing Lu Shizhe Diao Ellie Evans Jiaqi Zeng Pavlo Molchanov Yejin Choi Jan Kautz Yi Dong Collaborators: Vivienne Zhang Isabel Hulseman, MBA Seph Mard Pablo Ribalta, PhD Grzegorz Chlebus Wojciech Prazuch Ankit J. Patel

200

3 Comments

Our team ByteBrain, in collaboration with Tsinghua University, proposed ChatTS, a time-series multimodal LLM, accepted to VLDB 2025. Here is a brief introduction: Understanding time series is crucial for its application in real-world scenarios. Recently, large language models (LLMs) have been increasingly applied to time series tasks, leveraging their strong language capabilities to enhance various applications. However, research on multimodal LLMs (MLLMs) for time series understanding and reasoning remains limited, primarily due to the scarcity of high-quality datasets that align time series with textual information. This paper introduces ChatTS, a novel MLLM designed for time series analysis. ChatTS treats time series as a modality, similar to how vision MLLMs process images, enabling it to perform both understanding and reasoning with time series. To address the scarcity of training data, we propose an attribute-based method for generating synthetic time series with detailed attribute descriptions. We further introduce Time Series Evol-Instruct, a novel approach that generates diverse time series Q&As, enhancing the model’s reasoning capabilities. To the best of our knowledge, ChatTS is the first MLLM that takes multivariate time series as input for understanding and reasoning, which is fine-tuned exclusively on synthetic datasets. We evaluate its performance using benchmark datasets with real-world data, including six alignment tasks and four reasoning tasks. Our results show that ChatTS significantly outperforms existing vision-based MLLMs (e.g., GPT-4o) and text/agent-based LLMs, achieving a 46.0% improvement in alignment tasks and a 25.8% improvement in reasoning tasks. We have open-sourced the source code, model checkpoint and datasets at https://lnkd.in/gYDJ9S4j Paper: https://lnkd.in/gqXW9tPY About ByteBrain Team: ByteBrain is a platform of AI for Infra (AI for System) at ByteDance. The goal is to leverage AI to automatically optimize system performance and reduce labor and resource costs. ByteBrain integrates machine learning, LLM and OR techniques to support both internal infrastructure systems at ByteDance as well as the Volcano Engine Cloud.

151

3 Comments

🚀 New Paper: Localizing Knowledge in Diffusion Transformers Understanding how knowledge is distributed across the layers of generative models is crucial for improving interpretability, controllability, and adaptation. While prior work has explored knowledge localization in UNet-based architectures, Diffusion Transformer (DiT)-based models remain underexplored in this context. In this paper, we propose a model- and knowledge-agnostic method to localize where specific types of knowledge are encoded within the DiT blocks. We evaluate our method on state-of-the-art DiT-based models, including FLUX, PixArt-alpha, and SANA, across six diverse knowledge categories. We show that the identified blocks are both interpretable and causally linked to the expression of knowledge in generated outputs. Building on these insights, we apply our localization framework to two key applications: model personalization and knowledge unlearning. In both settings, our localized fine-tuning approach enables efficient and targeted updates, reducing computational cost, improving task-specific performance, and better preserving general model behavior with minimal interference to unrelated or surrounding content. Overall, our findings offer new insights into the internal structure of DiTs and introduce a practical pathway for more interpretable, efficient, and controllable model editing. 🌐 Project page: https://lnkd.in/dFWi6gJ2 📄 Paper: https://lnkd.in/dhBEF5Da Joint work with: Samyadeep Basu, Keivan Rezaei, Zihao Lin, Sayan Nag, Soheil Feizi

159

23 Comments

🚀 Fast-dLLM v2: Parallel Block-Diffusion Decoding for LLMs ⚡️ Highlights 🌟   - Blockwise bidirectional context via complementary masks   - Hierarchical caches (block + sub-block)   - Parallel sub-block decoding + token-shift training Results 📊 - ~2.5× faster vs. standard AR decoding on A100   - 102.5 tok/s (bs=1), 201.0 tok/s (bs=4) — >2× Qwen2.5   - SOTA efficiency–quality trade-off among diffusion LLMs Paper/Code 🌐: https://lnkd.in/g3AhGp96

263

𝗟𝗼𝗥𝗔: 𝗠𝗮𝗸𝗶𝗻𝗴 𝗟𝗟𝗠 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 When we fine-tune a large language model, we’re essentially adjusting its weight matrices. The catch? These updates (ΔW) are just as large as the original weights (Wq). That means huge computational cost and massive storage overhead. Enter LoRA (Low-Rank Adaptation):  It starts with the idea that the actual changes we need (ΔW) don’t use the full dimensionality - they live in a lower-dimensional space. Instead of learning the entire ΔW, LoRA breaks it down into two much smaller matrices (B and A). So the update looks like this: W_new ≈ Wq + B × A  • Wq = frozen, pre-trained weights  • B & A = small trainable adapters --- Here’s why it matters:  • Huge parameter savings – A 4096×4096 weight matrix needs ~16M parameters for full tuning. With LoRA (rank=8), it drops to just 65K trainable params which is a 99.6% reduction.  • No inference slowdown – After training, LoRA’s small adapters are merged back into the model weights. So inference speed is identical to the original model.  • Easy task switching – Want a model tuned for sentiment, then switch to finance? Just load the tiny LoRA adapters, not the full model. In practice, LoRA is most often applied to the query & value matrices in the attention mechanism, giving strong results without touching the full model. --- Takeaway: This simple yet powerful trick is why LoRA has become the go-to method for parameter-efficient fine-tuning (PEFT). ♻️ Share it with your network if you find it useful, and follow Mayank Sultania for more practical AI tips. Original Paper for LoRA: Low-Rank Adaptation of Large Language Models -https://lnkd.in/gCGcgXSn Pdf by: Shirin P. #LoRA #LLM #AIEngineering #ParameterEfficient #GenAI

146

6 Comments

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

70

2 Comments

Excited to share about two manuscripts to be presented at ICASSP 2025! 🎉 1️⃣ Speech-N-LlaMA: Solving the Exposure Bias Problem in Speech LLMs 🤖 🤝 Research done with LlaMA Speech team, AI at Meta The challenge: LLMs are sequence-to-sequence networks that use teacher forcing (golden output) during training, but inference happens auto-regressively → creating an "exposure bias" gap! Our solution: N-pass decoding during training to bridge this gap Results: Up to 18% relative WER improvement with zero additional inference compute! 2️⃣ DiffSSD: Catching Up to Diffusion-Based Speech Generators 🤝 Done during and w/ PhD collaborators at Purdue University and ISPL - Politecnico di Milano The problem: Showed current synthetic speech detectors are falling behind! They don't generalize to recent diffusion-based generators Our solution: DiffSSD - a new 200-hours dataset featuring speech from both open-source and commercial diffusion-based generators Impact: Detectors trained with this dataset can counter synthetic speech from the latest speech generator tools ICASSP proceedings can be found in comment. Thanks to collaborators Gil, Desh, Ozlem, and all peers from LlaMA Speech team, Kratika, Prof. Delp, and Prof. Bestagini. #AI #SpeechLLMs #GenerativeAI #Deepfake #LLMs

177

5 Comments

Nvidia Dynamo Distributed Inference Framework: Rust mentioned btw. This project looks awesome as a tool for distributed inference. They’ve put a lot of thought into architecting this, with the ability to provision separate resources for disaggregated prefill and decode, dynamic GPU scheduling based on demand, LLM aware request routing, accelerated data transfer with a new data transfer library called nixl, and smart kv cache offloading. This framework comes in a modular format with a Python front end and a rust backend, so you get safety and fearless concurrency with the ease of Python and the ability to use the whole thing, or only the parts you need. Finally, it supports vLLM as an inference backend (which is my favorite inference library). This project is also fully open-source with an Apache2 license, discord, and an engaging community. I can’t wait to PoC this and share additional feedback! Check out the GitHub here: https://lnkd.in/eJSG9Gby #softwareengineering #gpus

52

The PyTorch Foundation is excited to welcome vLLM as a PyTorch Foundation-hosted project. vLLM is a high-throughput, memory-efficient inference and serving engine designed for LLMs. vLLM has always had a strong connection with the PyTorch project. It is deeply integrated into PyTorch, leveraging it as a unified interface to support a wide array of hardware backends. These include NVIDIA GPUs, AMD GPUs, Google Cloud TPUs, Intel Corporation GPUs, Intel CPUs, Intel Gaudi HPUs, and Amazon Web Services (AWS) Neuron, among others. This tight coupling with PyTorch ensures seamless compatibility and performance optimization across diverse hardware platforms. 📢 Read the full announcement: https://hubs.la/Q03lmH_F0 #vLLM #PyTorchFoundation #PyTorch #OpenSourceAI

754

7 Comments