Abhilash Majumder

I gave nine closed-source language models the brand-new JEE-Advanced 2025 paper—the toughest under-grad Physics-Chem-Math exam in India—and many of them out-scored the human AIR-1 (332/360)!! Why JEE-Advanced is a clean STEM benchmark - Fresh content → Paper was held on 18 May 2025, so it isn’t floating around training corpora. - No biology, no trivia → Pure PCM reasoning across 96 high-pressure questions. - Uncontaminated → GPQA and annual AIME are excellent, but their problems can leak into model training sets. JEE-Adv 2025 avoids that. Knowledge Cut-offs - OpenAI o3/o4-mini models → 1 Jun 24 - Gemini latest models → Jan 25 - Anthropic latest models → Mar 25 - JEE Advanced 2025 Paper date → 18 May 25 Only Gemini 2.5-Pro/Flash released after the exam date, so 7/9 models had zero chance to memorise these questions Implications - Deterministic PCM Q&A at JEE level is essentially **solved** for frontier models.   - Latency and unit-economics are now the real bottlenecks. - I'm expecting inference routers -- that hit a cheap, fast model first and fall back to a stronger model only when its confidence score is low -- to become the norm

665

68 Comments

Large Memory Models for Long-Context Reasoning This paper focuses on improving long-context reasoning with explicit memory mechanisms. It presents LM2, a Transformer-based architecture equipped with a dedicated memory module to enhance long-context reasoning, multi-hop inference, and numerical reasoning. LM2 outperforms Recurrent Memory Transformer (RMT) by 37.1% and a non-memory baseline (Llama-3.2) by 86.3% on memory-intensive benchmarks. Key components: • Memory-Augmented Transformer – LM2 integrates a memory module that acts as an explicit long-term storage system, interacting with input tokens via cross-attention and updating using forget, input, and output gates. • Hybrid Memory Pathway – Unlike previous architectures, LM2 preserves the original Transformer flow while adding a parallel memory pathway, ensuring generalizability across tasks. • Dynamic Memory Updates – The model selectively updates stored information, preventing degradation over long contexts while maintaining efficiency. https://lnkd.in/eJd6-5yJ

191

7 Comments

NVIDIA 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗱𝗿𝗼𝗽𝘀 𝗻𝗲𝘄 𝗽𝗮𝗽𝗲𝗿 𝗼𝗻 “𝗦𝗺𝗮𝗹𝗹 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 𝗮𝗿𝗲 𝘁𝗵𝗲 𝗙𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗔𝗜”. ⬇️ Everyone is chasing bigger models. But this paper argues the exact opposite: Most agent workloads don’t need 175B params — they need precision, speed, and control. 𝗧𝗵𝗲 𝗸𝗲𝘆 𝘁𝗮𝗸𝗲𝗮𝘄𝗮𝘆? Small Language Models (SLMs) are not only good enough — they’re better for the majority of agentic use cases. 𝗛𝗲𝗿𝗲 𝗶𝘀 𝗮 𝗾𝘂𝗶𝗰𝗸 𝘀𝘂𝗺𝗺𝗮𝗿𝘆 𝗼𝗳 𝘁𝗵𝗲 𝗸𝗲𝘆 𝗽𝗼𝗶𝗻𝘁𝘀: ⬇️ 1. SLMs can already match or beat 30–70B LLMs on task-specific reasoning  → From Phi-3 to DeepSeek Distill, we now have 2–9B models outperforming legacy LLMs with 10–70× faster inference. 2. Most agents just run repetitive, scoped tasks  → Parsing. Routing. Tool calls. Summaries. You don’t need an all-knowing LLM — you need a fast, fine-tuned SLM that gets the job done. 3. LLMs are economically unsustainable at scale  → They dominate cloud costs and energy use. SLMs offer massive savings in latency, memory, and operational overhead. 4. SLMs run on edge and consumer devices  → Tools like ChatRTX show real-time agents can live on laptops or embedded systems — without phoning home to a GPU cluster. 5. Heterogeneous agent stacks are the path forward  → Use LLMs sparingly for general reasoning. Let SLMs handle 80% of workflows. More modular. More efficient. More robust. 6. SLMs are easier to fine-tune and align  → Lower hallucination risk, tighter output control, and better format consistency. Perfect for tool-driven agent environments. More in the comments and the paper below to download — but I’ll say this now: This paper might age like gold for every team trying to ship serious agents in production. 𝗜 𝗲𝘅𝗽𝗹𝗼𝗿𝗲 𝘁𝗵𝗲𝘀𝗲 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁𝘀 𝗮𝗿𝗼𝘂𝗻𝗱 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀 — 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲𝘆 𝗺𝗲𝗮𝗻 𝗳𝗼𝗿 𝗿𝗲𝗮𝗹-𝘄𝗼𝗿𝗹𝗱 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀 — 𝗶𝗻 𝗺𝘆 𝘄𝗲𝗲𝗸𝗹𝘆 𝗻𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿. 𝗬𝗼𝘂 𝗰𝗮𝗻 𝘀𝘂𝗯𝘀𝗰𝗿𝗶𝗯𝗲 𝗵𝗲𝗿𝗲 𝗳𝗼𝗿 𝗳𝗿𝗲𝗲: https://lnkd.in/dbf74Y9E --- Read the full paper here: https://lnkd.in/gYyzgRzR

674

103 Comments

## 🚨 Nvidia ≠ India! You've likely seen headlines like: > "Nvidia is now bigger than India's economy!" But is that a fair comparison? Not at all. Here's why: ### 🧮 The Right Way to Compare: #### 🔸 Revenue (Annual Flow): - India's GDP (2025): ~$4.19 trillion - Nvidia's annual revenue (TTM): ~$148.5 billion - India's GDP is ~28× larger than Nvidia's annual revenue. #### 🔸 Valuation (Total Worth): - Nvidia's Market Cap (2025): ~$4.75 trillion - Revenue multiple: ~32× - India's Implied Economic Value at 32× GDP: $4.19T × 32 ≈ $134 trillion - That means India would be ~28× Nvidia in valuation if valued the same way. ### 🚧 GDP ≠ National Wealth: GDP measures annual output, not total assets. It excludes: - Land, minerals, infrastructure, heritage, government assets, etc. Nvidia's valuation includes both tangible and intangible assets and future earnings. ### 🌍 India's True Wealth: - National Wealth in 2020: ~$12.6 trillion (World Bank) - Projected by 2025: ~$16–20 trillion National wealth reflects real assets, far exceeding India's GDP. ### ✅ Bottom Line: - Revenue: India is ~28× Nvidia - Valuation (same multiple): India is ~28× Nvidia - Asset base: India's true wealth dwarfs headline GDP numbers 📌 Conclusion: Don't be fooled by catchy headlines. Nvidia isn't bigger than India—context and scale matter. --- 👉 Remember: Always dig deeper when comparing economies and companies. ⚠️ Don't undermine your country like that. 🚨 Stay alert. Don't let misleading posts sway your view. 🚀 Share if this made sense to you!

91

7 Comments

Tutorial on computing CLIP scores for assessing image / text pair quality. Really enjoyed reading this blog post and example code by Betty Le Dem. "I would suggest mastering Ray locally at first." I like this quote, and despite Ray being designed for scale, we really optimize single-machine experience & performance, since that's where everyone starts out. Datasets are full of low quality data, so curating / filtering data is one of the most important things you can do to improve quality of outcomes. https://lnkd.in/g5m_zK5J

104

1 Comment

🚀 Big news — NeMo Evaluator is now open source! 🧪 We’re thrilled to announce that NeMo Evaluator SDK — the engine behind NVIDIA’s internal model benchmarking — is now open sourced and available to everyone: 🔗 https://lnkd.in/e3yaznR4 NeMo Evaluator SDK is an open-source platform for robust, reproducible, and scalable evaluation of large language models. It lets you run hundreds of benchmarks across popular evaluation harnesses against any OpenAI-compatible model API, all from auditable Docker containers for trustworthy and reproducible results. NeMo Evaluator is built on four core principles: ✅ Reproducibility by Default: Every config, random seed, and dependency is captured automatically for repeatable results. ⚙️ Scale Anywhere: Run on your laptop, a Slurm cluster, or cloud backends like Lepton AI without changing your workflow. 📊 State-of-the-Art Benchmarking: Access 100+ benchmarks from 18 open-source harnesses, covering reasoning, math, code, safety, and more. 🧩 Extensible and Customizable: Add new benchmarks, integrate private datasets, or connect custom exporters to your existing MLOps tooling. This release is a major step toward transparent, standardized, and community-driven model evaluation. From here on, we’ll be building NeMo Evaluator in the open — and we’d love for you to join us: try it, break it, file issues, contribute, and help shape its future! ⭐️ Explore it here → https://lnkd.in/e3yaznR4 If you find it useful, don’t forget to star the repo and help us grow the community! #NVIDIA #NeMo #Evaluator #OpenSource #LLM #Evaluation #AIResearch #MLOps #Benchmarking

255

3 Comments

𝗠𝗬 𝗪𝗘𝗘𝗞 𝗜𝗡 𝗔𝗜: As I wrote this week here on LinkedIn, https://bit.ly/4hDPOIY, DeepSeek’s recent advances reinforce a key insight—bigger isn’t always better. By combining retrieval-augmented generation (RAG) with model distillation, the DeepSeek work signals a shift toward hybrid AI, where scale fuels breakthroughs and efficiency refines them into real-world applications. As AI's rapid evolution continued this week, we were all shown that smarter, more efficient, and more responsible AI is the winning formula. Here are some of the top AI stories that stood out: 𝗢𝘄𝗻𝗶𝗻𝗴 𝗔𝗜 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗡𝗲𝘄 𝗠𝗼𝗮𝘁: Alphabet is ramping up AI spending to $75 billion, while SoftBank and OpenAI’s joint venture in Japan reflects how companies are consolidating AI assets to maintain leadership. Meanwhile, OpenAI’s Gemini 2.0 release and a new open-source reasoning model trained for under $50 reinforce that both scale and efficiency will shape AI’s competitive future. https://bit.ly/3EtOABw https://bit.ly/3WU5yPR https://bit.ly/3Qc4wuE 𝗧𝗵𝗲 𝗘𝗻𝗲𝗿𝗴𝘆 𝗖𝗿𝗶𝘀𝗶𝘀 𝗶𝘀 𝗔𝗜’𝘀 𝗕𝗶𝗴𝗴𝗲𝘀𝘁 𝗕𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸: As AI’s compute demand surges, Big Tech is co-locating AI data centers with renewable energy projects, while nuclear and solar are being explored as next-gen AI power sources. With Europe’s data centers already consuming 3% of total power, the industry is reaching an inflection point—companies that can scale efficiently will outlast those that simply scale bigger. https://bit.ly/42M9BSe https://bit.ly/40NtplB 𝗔𝗜 𝗶𝗻 𝗜𝗻𝗱𝘂𝘀𝘁𝗿𝘆: 𝗙𝗿𝗼𝗺 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝘁𝗼 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: AI’s biggest impact isn’t coming from general models—it’s coming from industry-specific integrations. From pick-and-place robots transforming manufacturing to AI-powered trucking logistics optimizing supply chains, AI’s next evolution isn’t about parameter counts but how well AI adapts to specific use cases. https://bit.ly/4guypRR https://bit.ly/4gy3bsS 𝗔𝗜 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗶𝘀 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 𝗤𝘂𝗶𝗰𝗸𝗹𝘆: Google’s quiet removal of its AI ethics pledge around weapons and surveillance raises questions about how AI will be used in military and security applications. As this comes on the heels of the U.S. Copyright Office’s decision to grant copyrights to AI-assisted works, the governance debate is no longer theoretical—it’s happening in real time.  https://bit.ly/40OLZtr 𝗔𝗜’𝘀 𝗣𝘂𝗯𝗹𝗶𝗰 𝗣𝗲𝗿𝗰𝗲𝗽𝘁𝗶𝗼𝗻 𝗶𝘀 𝗦𝗵𝗶𝗳𝘁𝗶𝗻𝗴: OpenAI is launching its first-ever Super Bowl ad, while Google had to re-edit an AI-generated ad after a content blunder. AI is becoming mainstream, and as its presence grows, so will public expectations for transparency, reliability, and trust. https://bit.ly/42HcxiZ https://bit.ly/3CulSjz #Deepseek #AI #ArtificialIntelligence Hitachi Digital Hitachi #OpenAI #datacenter #RAG

46

So. DeepSeek R1. Yes, it is impressive. But few of the hot takes or news articles on it are meaningfully correct. *What is True* - DeepSeek is a CoT model that benchmarked similarly to recent ChatGPT o1. - Inference on DeepSeek is cheap. Raw tokens are 30x cheaper and overall question responses (balancing a tendency to be a bit more verbose than o1) are probably about 20x cheaper. That's amazing! If you believe that costs halving every quarter is a trend then that was a 13-18 month acceleration (buyer beware: many assumptions in that sentence). - Unless a closed-source lab creates frontier performance with (an unknown) novel architecture, we can expect the latest GenAI performance to rapidly distilled and copied into other models. You probably don't need the latest Nvidia cards to do that, but you do still need an incredible team. *What is not True* - It did not cost $6M to train, and saying "that was just the final run" isn't really helpful. DeepSeek's parent has spent $1.3B on building datacentres before salary & power costs, and they've been hiring grad talent at rumored $1M+ salaries. DeepSeek is cheaper than US labs but not unfathomably so. - DeepSeek didn't come out of nowhere as an AI Lab. Avid watchers of AI knew about that spending, and that mid-2024 DeepSeek were getting impressive results. - R1 is not the only model competitive with o1. I think DeepSeek is dominating the hype because it is a Chinese lab and because they co-launched the news with a free app (much like ChatGPTs explosion in Nov 2022) -- unlike Gemini Flash 2.0 Thinking. Also, o3 is known to significantly outperform o1. *Further details* - DeepSeek's parent was purchasing GPUs at scale for years before export restrictions and continue to buy aggressively. They probably have many 10's-of-thousands of a mix of H100, H800s, H20s, etc... (not 50k+ H100s. 50k Hopper-generation cards) - DeepSeek has made huge changes in code and architecture to make training more efficient and inference cheaper. I absolutely think it is possible to train DeepSeek on lower-end Nvidia hardware that is legal based on export restrictions. There are many bad takes out there, but at a minimum I'd not listen to anyone who can't explain why CUDA and PTX are distinct and how PTX code optimized DeepSeek and worked around the limitations of lower-end GPUs. - Important architecture improvements were published starting in Jan 2024: Multi-head Latent Attention (MLA), DeepSeekMoE, Auxiliary-Loss-Free Load Balancing. All are currently unique to DeepSeek but are, I expect, being copied. Roughly speaking, the common theme is getting smarter about using less data and activating only the right parts of a network to answer a question. - We still don't exactly know the evidence for o1 distillation, but it is interesting that Microsoft is claiming some credit -- observation of calls to gateway perhaps?

33

📢 New blog post: Reinforcement Learning with NVIDIA NeMo‑RL – Reproducing the DeepScaleR Recipe with GRPO 🚀 We just published a deep‑dive showing how NeMo‑RL, our new open‑source post‑training library in the NVIDIA NeMo framework, scalably trains reasoning models from a single GPU prototype to thousand‑GPU clusters. In the tutorial we: ⚙️ Walk through NeMo‑RL’s flexible, backend‑agnostic design (HF, vLLM today; Megatron‑Core & TensorRT‑LLM coming) 🔄 Demonstrate Group Relative Policy Optimization (GRPO) on the DeepScaleR‑1.5B recipe 📈 Reach AIME 2024 scores that surpass OpenAI O1—in just a few hundred training steps 🛠️ Provide step‑by‑step scripts, configs, and evaluation tools so you can reproduce results out‑of‑the‑box If you’re exploring RLHF/RLAIF, curriculum scaling, or efficient long‑context training, this post is for you. 👉 Read the blog and grab the code: https://lnkd.in/gPh5n_SZ #NVIDIA #NeMo #ReinforcementLearning #GRPO #DeepScaleR #OpenSource #AIResearch

41

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

70

2 Comments

Excited to share two new blog posts I co-authored with my colleague Giuseppe on optimizing Large Language Model (LLM) training with NVIDIA Grace Hopper! 🚀 Together, we dove deep into practical strategies and tools to help you get the most out of your LLM training workflows: 1️⃣ Profiling LLM Training Workflows We walk through how to use NVIDIA Nsight Systems to pinpoint bottlenecks and optimize both CPU and GPU utilization. https://lnkd.in/d5G-MtpB 2️⃣ Advanced Optimization Strategies Explore advanced techniques like CPU offloading, Unified Memory, AMP, and FP8 training to push performance even further. https://lnkd.in/dngHYQDU If you’re working with LLMs on Grace Hopper, we hope these resources help you unlock new levels of efficiency and scalability. #AI #NVIDIA #LLM #GraceHopper #DeepLearning #Optimization

85

1 Comment

🚀 Fast-dLLM v2: Parallel Block-Diffusion Decoding for LLMs ⚡️ Highlights 🌟   - Blockwise bidirectional context via complementary masks   - Hierarchical caches (block + sub-block)   - Parallel sub-block decoding + token-shift training Results 📊 - ~2.5× faster vs. standard AR decoding on A100   - 102.5 tok/s (bs=1), 201.0 tok/s (bs=4) — >2× Qwen2.5   - SOTA efficiency–quality trade-off among diffusion LLMs Paper/Code 🌐: https://lnkd.in/g3AhGp96

263

🚀 Advances in Evaluating Large Language Models – April 2025 Overview As LLMs become central to enterprise, healthcare, and research applications, rigorous evaluation is no longer optional—it's foundational. Pun Intended !! :) In my latest piece, I dive into the evolving science and engineering of LLM evaluation. I synthesize recent breakthroughs across: 🔍 Lifecycle observability & instrumentation ⚠️ Error analysis & failure mode detection 🧠 Human-in-the-loop evaluation 📊 Cost-performance trade-offs 📚 And more—complete with links to must-read research papers and abstracts Whether you're building, deploying, or governing LLM systems, this article offers a roadmap to think more critically about performance, safety, and reliability. 📝 Read the full article here: https://lnkd.in/gWzc6iKk Let’s shape a future where language models are not just powerful—but also transparent, accountable, and aligned with human values. #AI #LLM #MachineLearning #ResponsibleAI #LLMEvaluation #AIGovernance #Observability #ErrorAnalysis #AIResearch 

41

2 Comments

NVFP4 is awesome 😎 but how do you get your models there? 🤔 We cover that and more in our latest tech deep-dive on TensorRT's Model Optimizer post-training quantization: https://lnkd.in/eMtAMvAa I've included some jupyter notebooks to help you get started on NVFP4/FP8, Activation-Aware Weight Quantization (AWQ), and AutoQuantization: https://lnkd.in/ebwb7nbH If you're not optimizing your models using the right tools, you're leaving critical performance on the table. Learn more about NVFP4: https://lnkd.in/dFrdpiXv #ai #machinelearning #deeplearning #llms #genai #nvfp4 #nvidia #blackwell #developers

52

2 Comments