Moin Nabi

The most cited paper of the 21st century is on deep residual learning with residual connections. Who invented this? Timeline: ★ 1991: Sepp Hochreiter introduced residual connections for recurrent neural networks (NNs) in a diploma thesis supervised by J. Schmidhuber, at a time when compute was about 10 million times more expensive than today (2025). His recurrent residual connection (weight 1.0) for a neural unit with the identity activation function was mathematically derived from first principles to solve the fundamental deep learning problem of vanishing or exploding gradients, first identified and analyzed in the very same thesis. ★ 1997 LSTM: Recurrent residual connections (see above) are a defining feature of what was called Long Short-Term Memory (LSTM) in a 1995 tech report. The subsequent LSTM journal paper (1997) has become the most cited AI paper of the 20th century. The LSTM core units with residual connections (weight 1.0) were called constant error carrousels (CECs). ★ 1999 LSTM: gated recurrent residual connections (gates initially open: 1.0). The 1999 LSTM could learn when to close those gates. This reintroduces the vanishing gradient problem [VAN1], but in a controlled way. With PhD student Felix Gers and postdoc Fred Cummins. ★ 2005: unfolding LSTM—from recurrent to feedforward residual NNs. With PhD student Alex Graves. ★ May 2015: very deep Highway Net with gated feedforward residual connections (initially 1.0). With PhD students Rupesh Kumar Srivastava and Klaus Greff. First gradient-based feedforward NN (FNN) with hundreds of layers, 10 times deeper than previous FNNs. ★ Dec 2015: ResNet — like an open-gated Highway Net (or an unfolded 1997 LSTM). Microsoft's ResNet paper refers to the Highway Net as 'concurrent'. However, this is incorrect: ResNet was published seven months later. The ResNet paper acknowledges the problem of vanishing/exploding gradients, but fails to recognise that S. Hochreiter first identified the issue in 1991 and developed the residual connection solution (weight 1.0). The ResNet paper cites the earlier Highway Net in a way that does not make it clear that ResNets are essentially open-gated Highway Nets and that Highway Nets are gated ResNets. It also fails to mention that the gates of residual connections in Highway Nets are initially open (1.0), meaning that Highway Nets start out with standard residual connections, to achieve deep residual learning. The residual parts of a Highway Net are like those of an unfolded 1999 LSTM, while the residual parts of a ResNet are like those of an unfolded 1997 LSTM. A follow-up paper by the ResNet authors was flawed in its design, leading to incorrect conclusions about gated residual connections. Details and 100 references in: J. Schmidhuber. Who invented deep residual learning? Technical Report IDSIA-09-25, IDSIA, September 2025.  https://lnkd.in/e8bc6vNi  https://lnkd.in/eeVrH2T5

454

11 Kommentare

We are pleased to announce that our paper entitled "Implementation and verification of coherent error suppression using randomized compiling for Grover's algorithm on a trapped-ion device" has recently been released. This is a collaborative study with Prof. Emerson and Dr. Qassim at Keysight Technologies. In this paper, we demonstrate the effectiveness of randomized compiling (RC) and algorithm error detection (ED), where the latter is realized via post-selection on ancillary qubits that ideally return to the ground state at the end of each circuit. Our results highlight a synergetic effect: combining RC and ED yields the largest reductions in errors, indicating that these methods can work together to extend the capabilities of near-term quantum devices for moderately deep circuits. #Fujitsu #FujitsuResearch #Keysight #Quantum https://lnkd.in/gPd-jJKd

103

🚀𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗟𝗠 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 We’re excited to share our latest work on improving inference-time efficiency for LLMs through KV cache quantization—a critical step toward making long-context reasoning more scalable and memory-efficient. 🧠𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹𝘀 & 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲-𝘁𝗶𝗺𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 Modern reasoning models often require long responses to “think” through problems before arriving at a final answer. Inference-time scaling methods make this even more compute-intensive. While these approaches improve model performance, they incur higher latency and demand more GPU memory. 💡𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 The KV cache stores the intermediate representations from previous tokens to accelerate autoregressive decoding. Think of it as the model’s short-term memory—just as humans recall previous parts of a conversation to respond quickly, the KV cache helps models maintain and build on prior context. For long sequences, the KV cache can consume more GPU memory than the model weights. During inference, LLM decoding becomes memory-bound, with most of the time spent on data transfer rather than computation. This has led to active research on KV cache quantization but quantization errors can accumulate as more tokens are generated, causing later tokens to deviate from expected outputs. ✨𝗪𝗵𝗮𝘁’𝘀 𝗻𝗲𝘄 𝗶𝗻 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? We introduce 𝚂̲𝚀̲𝚞̲𝚊̲𝚝̲ (Subspace-orthogonal KV cache quantization)—a new method that significantly reduces memory overhead and latency while maintaining model accuracy. SQuat constructs a subspace that captures critical task-relevant information, then enforces quantization errors to lie orthogonal to this subspace, minimizing their effect on the output of the attention mechanism. 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: ✅Training-free: No fine-tuning or calibration data needed ✅On-the-fly: Runs during inference without modifying the model ✅Theory-grounded: Built on a theoretical foundation we developed ⚡𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁: • Reduces GPU peak memory by 2.17× to 2.82× • Improves throughput by 2.45× to 3.60× • outperform existing KV cache quantization methods on benchmark tasks 📄paper: https://lnkd.in/emKhAVZu 💻 code: https://lnkd.in/e8TJ7N3R 👏 Joint work with my amazing co-authors Ligong Han, Kai Xu, Akash Srivastava, at the Red Hat AI innovation team (https://lnkd.in/exd6QDbk) Chris Wright, Ruchir Puri, Steven Huels, Joe Fernandes, Tushar Katarki, Ritika Gunnar, Jason McGee, Máirín Duffy, Luke Inglis, Mark Kurtz, Nick Hill, Tyler Michael Smith, vLLM 

206

1 Kommentar

Excited to share two new blog posts I co-authored with my colleague Giuseppe on optimizing Large Language Model (LLM) training with NVIDIA Grace Hopper! 🚀 Together, we dove deep into practical strategies and tools to help you get the most out of your LLM training workflows: 1️⃣ Profiling LLM Training Workflows We walk through how to use NVIDIA Nsight Systems to pinpoint bottlenecks and optimize both CPU and GPU utilization. https://lnkd.in/d5G-MtpB 2️⃣ Advanced Optimization Strategies Explore advanced techniques like CPU offloading, Unified Memory, AMP, and FP8 training to push performance even further. https://lnkd.in/dngHYQDU If you’re working with LLMs on Grace Hopper, we hope these resources help you unlock new levels of efficiency and scalability. #AI #NVIDIA #LLM #GraceHopper #DeepLearning #Optimization

85

1 Kommentar

A TechCrunch article about a main-stage panel in Davos. Quote: "“I think the shelf life of the current [LLM] paradigm is fairly short, probably three to five years,” LeCun said. “I think within five years, nobody in their right mind would use them anymore, at least not as the central component of an AI system. I think [….] we’re going to see the emergence of a new paradigm for AI architectures, which may not have the limitations of current AI systems.” These “limitations” inhibit truly intelligent behavior in machines, LeCun says. This is down to four key reasons: a lack of understanding of the physical world; a lack of persistent memory; a lack of reasoning; and a lack of complex planning capabilities. “LLMs really are not capable of any of this,” LeCun said. “So there’s going to be another revolution of AI over the next few years. We may have to change the name of it, because it’s probably not going to be generative in the sense that we understand it today.” https://lnkd.in/emnCfZjq

2.597

179 Kommentare

Excellent news from FSE and MSR conferences. We have three accepted papers. 1. 𝗙𝗦𝗘 (𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝘁𝗿𝗮𝗰𝗸) - "𝘈𝘯 𝘢𝘥𝘢𝘱𝘵𝘪𝘷𝘦 𝘭𝘢𝘯𝘨𝘶𝘢𝘨𝘦-𝘢𝘨𝘯𝘰𝘴𝘵𝘪𝘤 𝘱𝘳𝘶𝘯𝘪𝘯𝘨 𝘮𝘦𝘵𝘩𝘰𝘥 𝘧𝘰𝘳 𝘨𝘳𝘦𝘦𝘯𝘦𝘳 𝘭𝘢𝘯𝘨𝘶𝘢𝘨𝘦 𝘮𝘰𝘥𝘦𝘭𝘴 𝘧𝘰𝘳 𝘤𝘰𝘥𝘦". It offers a pruning method maintaining ~original accuracy with substantially less computation for LMs. Authors: Mootez SAAD, Percy Chen, José Antonio Hernández López, Daniel Varro, and yt. Preprint: https://lnkd.in/ghQSMP_T 2. 𝗠𝗦𝗥 (𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝘁𝗿𝗮𝗰𝗸) - "𝘐𝘵 𝘞𝘰𝘳𝘬𝘴 (𝘰𝘯𝘭𝘺) 𝘰𝘯 𝘔𝘺 𝘔𝘢𝘤𝘩𝘪𝘯𝘦: 𝘈 𝘚𝘵𝘶𝘥𝘺 𝘰𝘯 𝘙𝘦𝘱𝘳𝘰𝘥𝘶𝘤𝘪𝘣𝘪𝘭𝘪𝘵𝘺 𝘚𝘮𝘦𝘭𝘭𝘴 𝘪𝘯 𝘈𝘯𝘴𝘪𝘣𝘭𝘦 𝘚𝘤𝘳𝘪𝘱𝘵𝘴". It offers a) a catalog of reproducibility smells, b) a tool REDUSE to identify these smells in Ansible code, and c) an empirical analysis. Authors: Ghazal Sobhani, Israat Haque, and yt. preprint: https://lnkd.in/gpbXG74A 3. 𝗠𝗦𝗥 (𝗧𝗼𝗼𝗹𝘀 𝘁𝗿𝗮𝗰𝗸) - "𝘋𝘗𝘺: 𝘊𝘰𝘥𝘦 𝘚𝘮𝘦𝘭𝘭𝘴 𝘋𝘦𝘵𝘦𝘤𝘵𝘪𝘰𝘯 𝘛𝘰𝘰𝘭 𝘧𝘰𝘳 𝘗𝘺𝘵𝘩𝘰𝘯". Authors: Aryan Boloori and yt. Preprint: https://lnkd.in/gDKhXMpN Dalhousie Faculty of Computer Science #software-engineering #ML4SE #AI

110

8 Kommentare

𝗪𝗵𝘆 𝗠𝗮𝗽𝗕𝗲𝗻𝗰𝗵 𝗶𝘀 𝗮 𝗚𝗮𝗺𝗲-𝗖𝗵𝗮𝗻𝗴𝗲𝗿: • 📌 1600+ Complex Pathfinding Queries from 100 uniquely challenging map scenarios (urban areas, theme parks, universities, malls, and more). • 📌 Introduces Map Space Scene Graph (MSSG): a novel data structure for mapping visual landmarks and spatial relationships to structured navigation tasks. • 📌 Evaluates state-of-the-art LVLMs like GPT-4o, Llama-3.2, and Qwen-2-VL under zero-shot and Chain-of-Thought (CoT) reasoning methods, revealing key insights into their spatial reasoning and navigation abilities. 🚩 𝗞𝗲𝘆 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀: • Despite their impressive capabilities, current LVLMs struggle significantly with spatial reasoning and structured decision-making. • CoT prompting boosts spatial reasoning performance but sometimes introduces redundant details. 👀 𝗖𝗵𝗲𝗰𝗸 𝗼𝘂𝘁 𝗼𝘂𝗿 𝗳𝗶𝗻𝗱𝗶𝗻𝗴𝘀, 𝗱𝗮𝘁𝗮𝘀𝗲𝘁, 𝗮𝗻𝗱 𝗰𝗼𝗱𝗲 𝗵𝗲𝗿𝗲: 🔗 Arxiv: lnkd.in/gBv-sFJ3 Via Zhengzhong Tu

56

2 Kommentare

2024 was a great year for building general-purpose LLMs, with some specialized fine-tuning for math and code. So far, 2025 seems to be the year of diverging into two key areas: (1) Reasoning models (focused on math and code), and (2) Agents (essentially LLM-based workflow automation). This year is going to be another eventful one for LLM research and development! I plan to write more about reasoning models soon. In the meantime, if you're looking for focused reading this weekend, I've re-compiled my take on the noteworthy LLM research papers of 2024. The topics include mixture-of-experts, new scaling laws for precision, scaling inference-time compute, and more. It's all packed into a PDF-friendly, 47-page article with a table of contents for easy navigation: https://lnkd.in/gFUT9cnk Topics: 1. January: Mixtral’s Mixture of Experts Approach  1.1 Understanding MoE models  1.2 The relevance of MoE models today 2. February: Weight-decomposed LoRA  2.2 LoRA Recap  2.2 From LoRA to DoRA  2.3 The future of LoRA and LoRA-like methods 3. March: Tips for Continually Pretraining LLMs  3.1 Simple techniques work  3.2 Will these simple techniques continue to work? 4. April: DPO or PPO for LLM alignment, or both?  4.1 RLHF-PPO and DPO: What Are They?  4.2 PPO Typically Outperforms DPO  4.3 How are PPO and DPO used today? 5. May: LoRA learns less and forgets less  5.1 LoRA learns less  5.2 LoRA forgets less  5.3 The LoRA trade-off  5.4 Future ​approaches to finetuning LLMs 6. June: The 15 Trillion Token FineWeb Dataset  6.1 Comparison to other datasets  6.2 Principled dataset development  6.3 The relevance of FineWeb today 7. July: The Llama 3 Herd of Models  7.1 Llama 3 architecture summary  7.2 Llama 3 training  7.3 Multimodal Llamas  7.4 Llama 3 impact and usage 8. August: Improving LLMs by scaling inference-time compute  8.1 Improve outputs by using more test-time computation  8.2 Optimizing test-time computation techniques  8.3 Test-time computation versus pretraining a larger model  8.4 Future relevance of test-time compute scaling 9. September: Comparing multimodal LLM paradigms  9.1 Multimodal LLM paradigms  9.2 Nvidia’s hybrid approach  9.3 Multimodal LLMs in 2025 10. October: Replicating OpenAI O1’s reasoning capabilities  10.1 Shortcut learning vs journey learning  10.2 Constructing long thoughts  10.3 Distillation – the quick fix?  10.4 The state of AI research  10.5 The future of LLMs in the light of o1 (and o3) 11. November: LLM scaling laws for precision  11.1 Chinchilla scaling laws refresher  11.2 Low-precision training  11.3 Precision scaling laws takeaways  11.4 Model scaling laws in 2025 12. December: Phi-4 and learning from synthetic data  12.1 Phi-4 performance  12.2 Synthetic data learnings  12.4 Future importance of synthetic data Conclusions and outlook  Multimodal LLMs  Computational efficiency  State space models  Scaling  What I am looking forward to

1.490

24 Kommentare

Mixed curvature models have had a lot of success in graph neural network training, where local embeddings account for local curvature. This paper applies mixed curvature techniques to knowledge graph embeddings and Tucker decompositions (a favorite tool from psychometrics) to fill in knowledge gaps in knowledge graphs. https://lnkd.in/euM9z839

65

5 Kommentare

What's the path from embedded #OLAP (DuckDB, Apache DataFusion) to #FaaS? Two years ago we shared the (first?) reference implementation of #duckdb as ephemeral engine over decoupled compute (#lambda): a good start for sure, but real-world details were left as exercises for the reader. With a bit of delay (sorry, busy building bauplan!), we are now releasing other pieces of the puzzle. Last year, we shared our Apache Iceberg + Apache Arrow cache with Ryan Curtin, and we now published how we hacked #duckdb planner to obtain a new type of intermediate representation for #lakehouse use cases. The result is a truly composable system: better I/O than "read_parquet" (which, especially 20 months ago, was not #S3 optimized), and a modular system in which engine, cache and storage format can be replaced. Our original Towards Data Science story (with code) is still online (https://lnkd.in/eyyDU3KR) and our pre-print is now available: https://lnkd.in/eyS6wrkD   If you are passionate about these topics, do not miss the CDMS@VLDB workshop on composable data systems by Satya Valluri, Mohamed Zait, Pedro Pedreira (et al.)! See you, #composable cowboys!

53

2 Kommentare

✨ Exciting insights from Yann Dubois's recent lecture at Stanford University on building Large Language Models (#LLMs) 🟢 Components of LLMs: Focus on architecture, training algorithms, data collection, evaluation methods, and system optimization for efficient model deployment. 🟢 Pre-training vs. Post-training: Understand the classical language modeling paradigm and the emerging trend of fine-tuning models like ChatGPT to function as AI assistants. 🟢 Importance of Tokenization: Effective tokenization strategies are crucial for handling diverse languages and ensuring model performance. 💎 This lecture is a must-watch for anyone interested in #AI advancements! 👉 https://lnkd.in/dcevp3cJ

44

1 Kommentar

Hey All! I am glad to be back with my 20th article in my series "Neural Networks Intuitions" and this time I will be talking about a 2024 released paper "Precision at Scale: Domain Specific Datasets On-demand", which outlines an interesting and an effective approach to discovery of concepts and data curation at scale, with minimal-to-zero human supervision, using foundation models such as VLMs and Diffusion models. Making use of foundation or large generic models to pseudo-label your dataset has been a widely adopted method to quickly get data tagged for any task at hand. And with more and more powerful & accurate models coming up, this task becomes simpler as the labels get less noisy and more accurate. The authors exploit this very fact and build a multi-stage solution for getting data labelled at scale. More specifically, I really like the authors' ideas to exploit LLMs/VLMs at every step(in every way possible) during the process of data curation. On a high level, they talk about: 1. How to adopt LLMs for concept discovery for any classification task? 2. How to pseudo label images with the help of image-text models? 3. OOD detection using CLIP 4. Synthetic data generation using Diffusion models. A common question(and a valid one) that arises is that why would anyone need domain specific data curation when these VLMs are getting better and more generalized on a regular basis. But making them work for any custom usecase that contains task specific, fine-grained patterns is the tricky part. Hence I strongly believe that such methods bridge the gap in building smaller, but accurate custom models as compared to their large counterparts. As a matter of fact, it also enables these general LLMs to be adapted/trained much quicker to such new domains(shorter feedback cycle). Do read the article to know more about the approach. Share your thoughts/comments as well. Cheers! https://lnkd.in/gxfbQwsr #neuralnetworks #largelanguagemodels #visionlanguagemodels #diffusion

41