Aditya Agarwal

I gave nine closed-source language models the brand-new JEE-Advanced 2025 paper—the toughest under-grad Physics-Chem-Math exam in India—and many of them out-scored the human AIR-1 (332/360)!! Why JEE-Advanced is a clean STEM benchmark - Fresh content → Paper was held on 18 May 2025, so it isn’t floating around training corpora. - No biology, no trivia → Pure PCM reasoning across 96 high-pressure questions. - Uncontaminated → GPQA and annual AIME are excellent, but their problems can leak into model training sets. JEE-Adv 2025 avoids that. Knowledge Cut-offs - OpenAI o3/o4-mini models → 1 Jun 24 - Gemini latest models → Jan 25 - Anthropic latest models → Mar 25 - JEE Advanced 2025 Paper date → 18 May 25 Only Gemini 2.5-Pro/Flash released after the exam date, so 7/9 models had zero chance to memorise these questions Implications - Deterministic PCM Q&A at JEE level is essentially **solved** for frontier models.   - Latency and unit-economics are now the real bottlenecks. - I'm expecting inference routers -- that hit a cheap, fast model first and fall back to a stronger model only when its confidence score is low -- to become the norm

665

68 Comments

DeepSeek recently published a paper on "Native Sparse Attention". Here is why it matters. The attention mechanism is core to the transformer architecture, which in turn is a building block for today's LLMs. Unlike recurrent networks, the attention mechanism can find causal relationships even in long inputs. Simply put, if you feed a 10,000 word book to a recurrent network, it is heavily compressed. Some of the causal relationships in input are lost. Example: Why did Harry Potter get his scar? This long range dependency could be lost. The attention mechanism is able to avoid this. It takes the entire input sequence together, and computes their relationships together, conserving long-range dependencies. DeepSeek has just published how they sped up this mechanism, using GPU-specific optimizations. Instead of computing attention in a dense manner (all-to-all O(N²) time complexity), DeepSeek uses SPARSE attention. Think of this as a sliding window of blocks, reducing time complexity to O(N*W). The unique contribution of this paper is the native support for this kind of sparse attention compute. DeepSeek is going down to the level of hardware optimisations to speed up training. Let me know your thoughts around this. Cheers! #AI #LLMs #DeepSeek

133

2 Comments

Some signs that Waymo has jumped on the end-to-end machine learning bandwagon. Junko Yoshida has a scoop from attending The Autonomous: signs that Waymo is doing tentative deployments of end-to-end machine learning in their fleet. Complete with a quote from Waymo that seems to support this (or at least does not deny it). Such a change has deep implications for scalability as well as safety. More about this in the article. I fully expect that Waymo will be smarter than a simplistic pixels-in/driving-out approach, as indicated in the story. But what that will look like is speculation at this point. https://lnkd.in/e4vTmRTs

88

9 Comments

🚀 Domain specific systems need custom AI system experimentations. Last week, I ran a fine-tuning experiment on Qwen2.5-Embedding-4B because sometimes those “off-the-shelf” big-boys models don’t quite capture domain nuance where nomenclature is a must. But here’s the thing 👇 Finetuning for domain specific problems isn’t just “load data → finetune model→ get results”. It requires deliberate expert choices: 1. The right loss function. contrastive? triplet? cosine similarity? 2. An evaluator that reflects real-world needs. 3. Carefully selected model that's usually domain-driven by choice (size vs speed vs domain fit) 4. Data strategies like in-batch negatives and efficient batching What this means in practice? → Higher MRR@10 → Better MAP@100 througout the pipelines → A stronger foundation for workloads and even multi-agent workflows The fun part is fine-tuning these smaller-but-smart models is no longer a “research only” luxury. With the right setup, anyone can spin up domain-specialized models that outperform bigger models on specific tasks. Do you build intelligent systems, How have you been doing this: are you sticking with general-purpose DL models, or are you already moving toward domain-tuned representations? In AI right now, the “fit” often beats the “size.” #AI #Embeddings #LLM #VectorSearch #MachineLearning PS. photo was from my time at #NAIS National AI Strategy workshop, Abuja

39

1 Comment

Vision Large Language Models (vLLMs) extend text-based LLMs to understand visual modalities like images and video. Here’s exactly how they work... Top-level view. vLLM architectures have two primary components: 1. LLM backbone: standard decoder-only transformer. 2. Vision encoder: usually a CLIP / ViT model (with optional Perceiver Resampler for handling videos). Visual tokens. The vision encoder takes an image (or video) as input and returns a fixed-size set of visual “token” vectors to represent this image as an output. Usually, we create visual tokens by sampling vectors from multiple layers of the CLIP model and concatenating them together, which ensures little perceptual information is lost. Unified embedding. Now, we have a set of text and image (or video) token vectors as input. The first common vLLM architecture simply: - Concatenates the two modalities of vectors together. - Passes these concatenated vectors as input to a decoder-only transformer. The size of the visual token vectors may not match that of the text token vectors, so we linearly project the visual token vectors into the correct dimension. The unified embedding architecture is conceptually simple, but it increases the length of input passed to the LLM, which increases computational costs during training / inference. These visual tokens are passed through every layer of our powerful LLM backbone! Cross-modality attention. Instead of concatenating text and vision token vectors, we just pass text token vectors as input to the LLM. To incorporate vision info, we can add cross-attention modules that perform cross-attention between the text and vision token vector into select layers of the LLM—usually every second or fourth layer. This architectural variant, which looks similar to the transformer decoder, is called the cross-modality attention architecture. This architecture merges visual info into the LLM using cross-attention (more efficient) instead of increasing input length. Additionally, it adds new layers into the model for fusing visual / text info, rather than relying on existing LLM layers. So, we can leave the LLM backbone fixed during training and only train the added layers, ensuring the LLM’s performance on text-only tasks stays the same. Training vLLMs. vLLMs are trained similarly to any other LLM–using next token prediction. There are two strategies for training a vLLM like this: 1. Natively multi-modal: train the model from scratch using multi-modal data from the beginning and throughout the entire training process. 2. Compositional: first pretrain the vision encoder / LLM backbone separately, then perform a separate training stage to fuse them together. Many examples of natively mutli-modal LLMs now exist (e.g., Gemini or GPT-4o). However, natively multi-modal training can be complex, and compositional training has several benefits (e.g., model development can be parallelized, we can leverage existing text-based LLMs, etc.).

419

25 Comments

Why 2025 is the tipping-point for Machine Learning Systems Engineering I just finished working through Machine Learning Systems by Prof. Vijay Janapa Reddi and team, and a few insights really crystallized for me: 1. From algorithms to engineered systems – success in AI now hinges as much on disciplined engineering as on model innovation. We’re witnessing the birth of a new discipline—Machine Learning Systems Engineering—that unites hardware, software and data to deliver reliable, scalable AI solutions. 2. AI = Data + Algorithms + Compute – a production ML system is an integrated triangle; ignore any corner and the whole structure collapses. 3. Efficiency isn’t optional – whether you’re running models in the cloud, on the edge, or on micro-controllers, resource-aware design (compression, accelerator-aware architectures, energy-savvy ops) is now table-stakes. 4. Emerging trends – agentic systems, heterogeneous accelerators, and sustainability-driven optimisation are reshaping every layer of the stack. Staying ahead means designing for adaptability and technical-debt resilience from day one. The textbook is open-source (<https://lnkd.in/dP9YEitq>), packed with slides, labs and case studies—from Microsoft FarmBeats to AlphaFold—making it a must-read for engineers, researchers and product leaders alike. 🔗 Curious how your team is preparing for the shift from “models” to “systems”? Let’s discuss! #MachineLearning #AIEngineering #MLOps #EdgeAI #TinyML #SustainableAI #TechLeadership

35

1 Comment

Recent research by UIUC and Intel Labs has introduced a new jailbreak technique for Large Language Models (LLMs) known as InfoFlood. This method takes advantage of a vulnerability termed "Information Overload," where excessive linguistic complexity can circumvent safety mechanisms without the need for traditional adversarial prefixes or suffixes.   InfoFlood operates through a three-stage process: Linguistic Saturation, Rejection Analysis, and Saturation Refinement. Initially, it reformulates potentially harmful queries into more complex structures. If the first attempt does not succeed, the system analyzes the response to iteratively refine the query until a successful jailbreak is achieved. Empirical validation across four notable LLMs—GPT-4o, GPT-3.5-turbo, Gemini 2.0, and LLaMA 3.1—indicates that InfoFlood significantly surpasses existing methods, achieving success rates up to three times higher on various benchmarks.   The study underscores significant vulnerabilities in current AI safety measures, as widely used defenses, such as OpenAI’s Moderation API, proved ineffective against InfoFlood attacks. This situation raises important concerns regarding the robustness of AI alignment systems and highlights the necessity for more resilient safety interventions. As LLMs become increasingly integrated into diverse applications, addressing these vulnerabilities is crucial for ensuring the responsible deployment of AI technologies and enhancing their safety against emerging adversarial techniques.  Arxiv: https://lnkd.in/eBty6G7z

53

4 Comments

𝗠𝗬 𝗪𝗘𝗘𝗞 𝗜𝗡 𝗔𝗜: As I wrote this week here on LinkedIn, https://bit.ly/4hDPOIY, DeepSeek’s recent advances reinforce a key insight—bigger isn’t always better. By combining retrieval-augmented generation (RAG) with model distillation, the DeepSeek work signals a shift toward hybrid AI, where scale fuels breakthroughs and efficiency refines them into real-world applications. As AI's rapid evolution continued this week, we were all shown that smarter, more efficient, and more responsible AI is the winning formula. Here are some of the top AI stories that stood out: 𝗢𝘄𝗻𝗶𝗻𝗴 𝗔𝗜 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗡𝗲𝘄 𝗠𝗼𝗮��: Alphabet is ramping up AI spending to $75 billion, while SoftBank and OpenAI’s joint venture in Japan reflects how companies are consolidating AI assets to maintain leadership. Meanwhile, OpenAI’s Gemini 2.0 release and a new open-source reasoning model trained for under $50 reinforce that both scale and efficiency will shape AI’s competitive future. https://bit.ly/3EtOABw https://bit.ly/3WU5yPR https://bit.ly/3Qc4wuE 𝗧𝗵𝗲 𝗘𝗻𝗲𝗿𝗴𝘆 𝗖𝗿𝗶𝘀𝗶𝘀 𝗶𝘀 𝗔𝗜’𝘀 𝗕𝗶𝗴𝗴𝗲𝘀𝘁 𝗕𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸: As AI’s compute demand surges, Big Tech is co-locating AI data centers with renewable energy projects, while nuclear and solar are being explored as next-gen AI power sources. With Europe’s data centers already consuming 3% of total power, the industry is reaching an inflection point—companies that can scale efficiently will outlast those that simply scale bigger. https://bit.ly/42M9BSe https://bit.ly/40NtplB 𝗔𝗜 𝗶𝗻 𝗜𝗻𝗱𝘂𝘀𝘁𝗿𝘆: 𝗙𝗿𝗼𝗺 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝘁𝗼 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: AI’s biggest impact isn’t coming from general models—it’s coming from industry-specific integrations. From pick-and-place robots transforming manufacturing to AI-powered trucking logistics optimizing supply chains, AI’s next evolution isn’t about parameter counts but how well AI adapts to specific use cases. https://bit.ly/4guypRR https://bit.ly/4gy3bsS 𝗔𝗜 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗶𝘀 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 𝗤𝘂𝗶𝗰𝗸𝗹𝘆: Google’s quiet removal of its AI ethics pledge around weapons and surveillance raises questions about how AI will be used in military and security applications. As this comes on the heels of the U.S. Copyright Office’s decision to grant copyrights to AI-assisted works, the governance debate is no longer theoretical—it’s happening in real time.  https://bit.ly/40OLZtr 𝗔𝗜’𝘀 𝗣𝘂𝗯𝗹𝗶𝗰 𝗣𝗲𝗿𝗰𝗲𝗽𝘁𝗶𝗼𝗻 𝗶𝘀 𝗦𝗵𝗶𝗳𝘁𝗶𝗻𝗴: OpenAI is launching its first-ever Super Bowl ad, while Google had to re-edit an AI-generated ad after a content blunder. AI is becoming mainstream, and as its presence grows, so will public expectations for transparency, reliability, and trust. https://bit.ly/42HcxiZ https://bit.ly/3CulSjz #Deepseek #AI #ArtificialIntelligence Hitachi Digital Hitachi #OpenAI #datacenter #RAG

46

🚀 Advances in Evaluating Large Language Models – April 2025 Overview As LLMs become central to enterprise, healthcare, and research applications, rigorous evaluation is no longer optional—it's foundational. Pun Intended !! :) In my latest piece, I dive into the evolving science and engineering of LLM evaluation. I synthesize recent breakthroughs across: 🔍 Lifecycle observability & instrumentation ⚠️ Error analysis & failure mode detection 🧠 Human-in-the-loop evaluation 📊 Cost-performance trade-offs 📚 And more—complete with links to must-read research papers and abstracts Whether you're building, deploying, or governing LLM systems, this article offers a roadmap to think more critically about performance, safety, and reliability. 📝 Read the full article here: https://lnkd.in/gWzc6iKk Let’s shape a future where language models are not just powerful—but also transparent, accountable, and aligned with human values. #AI #LLM #MachineLearning #ResponsibleAI #LLMEvaluation #AIGovernance #Observability #ErrorAnalysis #AIResearch 

41

2 Comments

🤖 𝗟𝗮𝘂𝗻𝗰𝗵𝗶𝗻𝗴 Sereact 𝗖𝗼𝗿𝘁𝗲𝘅 🤖 Most robotics initiatives don’t fail because of algorithms — they fail because of missing data foundations. Simulation can only go so far. Lab datasets can’t capture the messiness of production. 𝐴𝑡 𝑆𝑒𝑟𝑒𝑎𝑐𝑡, 𝑤𝑒 𝑡𝑜𝑜𝑘 𝑎 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑡 𝑝𝑎𝑡ℎ. With Cortex, we’ve built on the largest corpus of real-world production data in robotics — now spanning more than 100 live systems across the globe. 🦾 Why Cortex is different: - Built on production, not lab data → intelligence grounded in reality. - One brain for any robot, any task → generalization across robots & tasks. - A compounding flywheel → every new deployment makes Cortex smarter. With every additional system, Cortex becomes more adaptive and resilient in the field. This is how we move from brittle automation to general robotic intelligence that scales. Super proud of the team for making this milestone real. Cortex is live. #AI #Robotics #VLA

1,231

72 Comments

For those diving deep into deploying LLMs benchmarking is key. Our teams at NVIDIA share their expertise to help get started with the fundamentals of benchmarking for both latency and throughput. https://lnkd.in/etZaXAXV #generativeAI #LLMbenchmarking #testtimecompute

51

5 Comments

2024 was a great year for building general-purpose LLMs, with some specialized fine-tuning for math and code. So far, 2025 seems to be the year of diverging into two key areas: (1) Reasoning models (focused on math and code), and (2) Agents (essentially LLM-based workflow automation). This year is going to be another eventful one for LLM research and development! I plan to write more about reasoning models soon. In the meantime, if you're looking for focused reading this weekend, I've re-compiled my take on the noteworthy LLM research papers of 2024. The topics include mixture-of-experts, new scaling laws for precision, scaling inference-time compute, and more. It's all packed into a PDF-friendly, 47-page article with a table of contents for easy navigation: https://lnkd.in/gFUT9cnk Topics: 1. January: Mixtral’s Mixture of Experts Approach  1.1 Understanding MoE models  1.2 The relevance of MoE models today 2. February: Weight-decomposed LoRA  2.2 LoRA Recap  2.2 From LoRA to DoRA  2.3 The future of LoRA and LoRA-like methods 3. March: Tips for Continually Pretraining LLMs  3.1 Simple techniques work  3.2 Will these simple techniques continue to work? 4. April: DPO or PPO for LLM alignment, or both?  4.1 RLHF-PPO and DPO: What Are They?  4.2 PPO Typically Outperforms DPO  4.3 How are PPO and DPO used today? 5. May: LoRA learns less and forgets less  5.1 LoRA learns less  5.2 LoRA forgets less  5.3 The LoRA trade-off  5.4 Future ​approaches to finetuning LLMs 6. June: The 15 Trillion Token FineWeb Dataset  6.1 Comparison to other datasets  6.2 Principled dataset development  6.3 The relevance of FineWeb today 7. July: The Llama 3 Herd of Models  7.1 Llama 3 architecture summary  7.2 Llama 3 training  7.3 Multimodal Llamas  7.4 Llama 3 impact and usage 8. August: Improving LLMs by scaling inference-time compute  8.1 Improve outputs by using more test-time computation  8.2 Optimizing test-time computation techniques  8.3 Test-time computation versus pretraining a larger model  8.4 Future relevance of test-time compute scaling 9. September: Comparing multimodal LLM paradigms  9.1 Multimodal LLM paradigms  9.2 Nvidia’s hybrid approach  9.3 Multimodal LLMs in 2025 10. October: Replicating OpenAI O1’s reasoning capabilities  10.1 Shortcut learning vs journey learning  10.2 Constructing long thoughts  10.3 Distillation – the quick fix?  10.4 The state of AI research  10.5 The future of LLMs in the light of o1 (and o3) 11. November: LLM scaling laws for precision  11.1 Chinchilla scaling laws refresher  11.2 Low-precision training  11.3 Precision scaling laws takeaways  11.4 Model scaling laws in 2025 12. December: Phi-4 and learning from synthetic data  12.1 Phi-4 performance  12.2 Synthetic data learnings  12.4 Future importance of synthetic data Conclusions and outlook  Multimodal LLMs  Computational efficiency  State space models  Scaling  What I am looking forward to

1,490

24 Comments

How far is Quantum Computing from being industrially useful? In this article, one of my last when I was at AWS, we explain how careful algorithmic design and error-mitigating middleware can help simulate continuum physics on Quantum hardware: https://lnkd.in/gChTpRS5

53

1 Comment

If you are someone pushing boundaries of LLM interaction, improving prompt engineering strategies, and most importantly building more robust AI application, this paper on agentic context engineering could be a good read for you !! https://lnkd.in/gG97xZGk

38

1 Comment

Meta and Yann LeCun were always right to bet on public research and open-weight LLMs. When OpenAI started grifting billions through hype and overpromising, open-source didn’t seem like the smart move. There was no real point in hiding the algorithms or the weights as it was inevitable that research would catch up. Even data isn’t much of a moat anymore, with the focus shifting toward reducing reliance on labeled datasets. Meta empowered researchers and gained massively in return. The third party research with the #LLaMA models proved how powerful distillation can be. #DeepSeek used the architecture to show that reinforcement learning struggles with small models, while distilled models preserve reasoning better. LLaMA is now the backbone for LoRA adapters and fine-tuning research. Consequently, Meta could confidently train the massive LLaMA 4 Behemoth and distill it into Scout and Maverick—Scout even running on a single H100 GPU. LLaMA 4 isn’t just great for its 10M-token context, multimodality, or coding tasks. It’s great because it will enable research on new fine-tuning methods for multimodality, quantisation etc, better LoRAs, and maybe a future without hallucinating RAG. And as researchers build on Meta’s models, Meta learns what works and focuses on actual innovation.

107

3 Comments

So. DeepSeek R1. Yes, it is impressive. But few of the hot takes or news articles on it are meaningfully correct. *What is True* - DeepSeek is a CoT model that benchmarked similarly to recent ChatGPT o1. - Inference on DeepSeek is cheap. Raw tokens are 30x cheaper and overall question responses (balancing a tendency to be a bit more verbose than o1) are probably about 20x cheaper. That's amazing! If you believe that costs halving every quarter is a trend then that was a 13-18 month acceleration (buyer beware: many assumptions in that sentence). - Unless a closed-source lab creates frontier performance with (an unknown) novel architecture, we can expect the latest GenAI performance to rapidly distilled and copied into other models. You probably don't need the latest Nvidia cards to do that, but you do still need an incredible team. *What is not True* - It did not cost $6M to train, and saying "that was just the final run" isn't really helpful. DeepSeek's parent has spent $1.3B on building datacentres before salary & power costs, and they've been hiring grad talent at rumored $1M+ salaries. DeepSeek is cheaper than US labs but not unfathomably so. - DeepSeek didn't come out of nowhere as an AI Lab. Avid watchers of AI knew about that spending, and that mid-2024 DeepSeek were getting impressive results. - R1 is not the only model competitive with o1. I think DeepSeek is dominating the hype because it is a Chinese lab and because they co-launched the news with a free app (much like ChatGPTs explosion in Nov 2022) -- unlike Gemini Flash 2.0 Thinking. Also, o3 is known to significantly outperform o1. *Further details* - DeepSeek's parent was purchasing GPUs at scale for years before export restrictions and continue to buy aggressively. They probably have many 10's-of-thousands of a mix of H100, H800s, H20s, etc... (not 50k+ H100s. 50k Hopper-generation cards) - DeepSeek has made huge changes in code and architecture to make training more efficient and inference cheaper. I absolutely think it is possible to train DeepSeek on lower-end Nvidia hardware that is legal based on export restrictions. There are many bad takes out there, but at a minimum I'd not listen to anyone who can't explain why CUDA and PTX are distinct and how PTX code optimized DeepSeek and worked around the limitations of lower-end GPUs. - Important architecture improvements were published starting in Jan 2024: Multi-head Latent Attention (MLA), DeepSeekMoE, Auxiliary-Loss-Free Load Balancing. All are currently unique to DeepSeek but are, I expect, being copied. Roughly speaking, the common theme is getting smarter about using less data and activating only the right parts of a network to answer a question. - We still don't exactly know the evidence for o1 distillation, but it is interesting that Microsoft is claiming some credit -- observation of calls to gateway perhaps?

33