Brian Spiering

Great survey on small language models. This 87-page survey closely examines small language models, defined as models sized between the minimum threshold for emergent abilities on specialized tasks and the maximum sustainable under resource constraints. SLMs are really worth keeping a close eye on. They are specifically useful in multi-agent systems.

168

6 Comments

🚀 Exciting News! Our New Paper on helping Multimodal LLMs see Beyond Language Prior got accepted to CVPR Workshop on Visual Concepts! 🚀 Ever wonder why Multimodal LLMs (MLLMs), despite seeing images, sometimes miss the bigger picture or rely too much on text? While text-only LLMs get rich feedback from every token, MLLMs often face a "sparse feedback" problem. They struggle to learn about image concepts not explicitly mentioned in text descriptions and can default to just predicting text based on language patterns, rather than truly seeing. 🔥 Our latest research tackles this head-on! We've developed novel training strategies that: 1️⃣ Deepen Visual Understanding using Visual Loss: Teach the MLLM to build a much richer internal representation of ALL visual concepts in an image. 2️⃣ Boost Visual Attention using Blank Tokens: Encourage the model to pay significantly more attention to what it sees by subtly weakening its over-reliance on predicting just from previous text tokens. 💡 The Impact? We're seeing strong performance improvements on demanding visual tasks in both upstream (core understanding) and downstream (application-level) settings! 🏆 Key Achievement: Our approach enables our Llama 3.1 8B based Llava model – which is smaller, uses a simpler model architecture, and operates on lower-resolution visual inputs – to MATCH the performance of the much larger Llama 3.2 11B on challenging visual reasoning benchmarks! 🤯 This shows the power of smarter training recipe! This is a crucial first step towards MLLMs that are more visually intelligent, reliable, and truly understand the world around them. Please stay tuned for our upcoming updates on model architecture advancements more conducive to capturing visual and language concepts as well as post training MLLMs to reason through tricky questions. 👉 Dive into the details! Read the full paper here: https://lnkd.in/gZcbw5yC A huge thank you to my incredible co-authors Aarti Ghatkesar Uddeshya Upadhyay and everyone who supported this journey! #MultimodalLLM #MLLM #VisualGrounding #VisCon

69

2 Comments

A candidate prepares for an NLP interview by diving deep into transformers. Attention heads. Positional encodings. Masked language modeling. First question? “What’s the difference between stemming and lemmatization?” Next: “Our sentiment model struggles with slang-heavy data. What would you try?” That’s how most NLP interviews actually go. They start with fundamentals, but then they move quickly to trade-offs and messy, real-world text. Can you explain why accuracy might be misleading? When would TF-IDF be enough? How would you debug model errors? Sounding advanced isn't even what it's about; more showing clear thinking under constraints. Check out a longer list of interview questions and answers here: https://ow.ly/M8Ob50YmnxN

26

🧠 LLMs are getting smarter—but their memory systems are still messy. Most papers tackle one slice of the problem (like retrieval or long-term storage). But what if we zoomed out? 📄 In this new survey, researchers propose a unified view of memory in AI: 🔹 A taxonomy of memory types (parametric, structured, unstructured) 🔹 A breakdown of six core operations (consolidation, retrieval, forgetting, and more) 🔹 And a mapping to real-world use cases across long-term reasoning, multi-source integration, and memory editing 🔍 Why it matters? Modern agents need memory to adapt, personalize, and plan. But without a clear operational framework, progress is fragmented. This paper connects the dots—with examples, benchmarks, and toolkits for building memory-centric systems. Whether you’re building long-term assistants or debugging memory faults in LLMs, this is a solid foundation to work from. #LLM #AI #Taxonomy #Operations #Topics #FutureDirections #RethinkingMemoryinAI

38

1 Comment

🧬 The concept of Neural DNA (nDNA), initially proposed by Dr. Amitava Das, opens up a new avenue for interpreting LLM behavior. nDNA treats an LLM as something we can sequence (just like DNA sequencing) by reading its internal signals -- its lineage and layer-level changes. It captures a model’s lineage, tuning-induced mutations, and functional expression so we can compare LLM families, spot drift, and explain why similar benchmarks behave differently across a range of LLMs. In practice, nDNA looks for signals such as: (i) layer shifts, (ii) MoE routing patterns, and (iii) controlled probes. It lets us audit change over time and pick models for tasks with evidence. Collaborating with Dr. Amitava Das, and his Pragya group (https://pragyaai.github.io) at the Department of CSIS BITS Pilani Goa Campus, we've been building and testing the concept of nDNA to understand LLM behavior across major job families: - Llama tends to refine mid–to-late layers while early layers remain stable; - Mistral localizes expertise while the early layers remain steady; - Gemma aligns with light-touch parameter changes; - Qwen rewrites more in its middle layers, especially in multilingual scenarios; - DeepSeek adds dialogue capability in mid layers, while preserving base geometry. One of the other distinct advantages of nDNA is the ability to identify model collapse. On the surface you might see a slow loss of creativity and nuance in a model’s behavior. Inside, through the nDNA lens, the model’s internal landscape flattens -- fewer distinct patterns light up -- much like reduced neural diversity in biology. With nDNA, that shift becomes measurable over time, which makes collapse diagnosable and monitorable. Dr. Amitava Das's novel thought process and insistence on clear hypotheses has led to the inception of nDNA as a practical lens for AI -- one that detects drift before deployment, keeps model lineages auditable, and matches models to tasks based on evidence rather than guesswork. Taken together, these nDNA lenses reframe how we interpret LLM behavior. Our other recent works, DPO-Kernels (ACL 2025; https://lnkd.in/gXGQvXwX), Yin-Yang Align (ACL 2025; https://lnkd.in/gWVCxG8B), QuickSilver (https://lnkd.in/gpZQKMmP), and the Counter Turing Test (EMNLP 2023 Outstanding Paper Award; https://lnkd.in/gaVHeXgJ), share a similar first-principle approach to LLM refinement/understanding. If nDNA is something you'd like to explore further -- we'll be launching a website for nDNA soon, stay tuned! #nDNA #AI #FoundationModels #ModelInterpretability

283

5 Comments

Many of us have faced last-minute issues while presenting slides. Missing fonts, media not playing, files incompatible with the host system, or simply differences between operating systems. These problems can throw you off in crucial moments like conferences, reviews, or lectures. To address this, I’ve started using Quarto with Reveal.js to create HTML-based presentations and host them on GitHub Pages. This makes my slides accessible from any browser. No need to carry files, no risk of version mismatches, and everything works exactly as intended. I’ve written a short blog post explaining how to set this up, for anyone who wants a smooth, reproducible way to present from anywhere. Blog Post: https://lnkd.in/d4cQuxpw Demo Slides: https://lnkd.in/dG-QqUHj Source Code: https://lnkd.in/d2kmzbrW Disclaimer: Slides hosted this way are publicly viewable, so it’s best suited for content you’re comfortable sharing openly.

39

Quantum needs data—not just qubits. aqora’s new public datasets hub marks a practical step forward for the field. By creating a central space to upload, access, and collaborate on quantum-specific datasets, they’re addressing a quiet but critical challenge in the ecosystem: reproducibility and shared benchmarks. Why does this matter? Quantum machine learning and benchmarking rely on consistent, high-quality data—but until now, much of it has been scattered, inconsistent, or locked in silos. This hub offers: ➡️ Versioned, well-structured datasets ➡️ Integration with tools like pandas and polars ➡️ Options for both public and private data sharing ➡️ A base for teaching, testing, and real-world use cases As the field matures, tools like this can help shift the conversation from isolated experiments to collaborative progress. 🔗 Full article in the comments. #QuantumComputing #QML #OpenData #QuantumResearch #TQI #TheQuantumInsider #Aqora #EcosystemBuilding

79

5 Comments

The article discusses new reranking model for question answering. Although it is designed for question answering over scientific literature, the underlying design principles are broadly applicable to other question-answering settings where semantic relevance, logical consistency, and evidential grounding are critical. https://lnkd.in/gyzQUaVa

17

Introducing Generative Adversarial Distillation (GAD): a novel GAN-style formulation and framework that facilitates both on-policy and black-box distillation of large language models (LLMs). GAD is the first technique to enable block-box on-policy distillation from proprietary teachers where internal logits or parameters are inaccessible, or distillation between teacher and student LLMs with incompatible vocabularies. GAD expands our prior work on white-box on-policy distillation (i.e., MiniLLM), pioneering block-box on-policy distillation for LLM training. Specifically, GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM’s, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation.  In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation. Our team has been conducting fundamental research in knowledge distillation with wide adoptions across the industry. -         MiniLM: We introduced multi-head attention distillation, establishing the most effective distillation method for BERT-style models. The open-source MiniLM models (e.g., 6x384) have become the most widely utilized small encoder models on the Hugging Face. -         MiniLLM: Our proposed Reverse KLD is recognized as one of the most effective, de facto on-policy distillation approaches for modern LLM training, which has been widely used by Thinking Machines, Gemma, and many other teams and models. -         BitDistill: We proposed BitNet Distillation to finetune off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (ternary weights {-1, 0, 1}), achieving performance parity with the full-precision counterparts on specific downstream tasks. -         GAD: The development of Generative Adversarial Distillation (GAD) now allows for black-box on-policy distillation, overcoming two major prior limitations: (1) Distillation from proprietary teachers where internal logits or parameters are inaccessible; (2) Distillation between teacher and student LLMs with incompatible vocabularies. https://lnkd.in/gMaP2c7w

125

2 Comments

📢 New Blog Post: NeurIPS Datasets & Benchmarks Track - From Art to Science in AI Evaluations we just published a new blog post that provides an update on the 2025 Datasets and Benchmarks Track, reflecting on how the new hosting and metadata requirements affected submissions and the review process. We present submission statistics, survey findings from 851 authors and 155 reviewers, and identify areas requiring continued development. This differs from our earlier post on the review process by focusing on empirical outcomes rather than procedural changes. with Konstantina Palla , Francesco Locatello , Joaquin Vanschoren , Megan Risdal https://lnkd.in/eHxQTvnh

93

AI on Review: How Large Language Models Are Reshaping Peer Review The Peer Review Crunch and New "Reviewer Duties" Peer review is the backbone of scientific quality control, ensuring that research findings are vetted for accuracy and significance before publication (pmc.ncbi.nlm.nih.gov). In fast-moving fields like machine learning and computer vision, top conferences function much like journals - and the integrity of science depends on rigorous peer evaluation. However, the system is straining under an avalanche of submissions. Major AI conferences now routinely receive well over 10,000 papers, a surge that has stretched the reviewer pool to its limits (arxiv.org). This deluge has led to radical policy changes: some conferences now essentially conscript all submitting authors into service as reviewers. For example, ICLR 2025 explicitly warned authors that any paper without at least one author signed up to review would be desk-rejected (reddit.com). NeurIPS and others have similarly pleaded that "all authors help with reviewing, if asked," to tackle the reviewer shortage. https://lnkd.in/d7KDP99K

26

Another interesting new RL perspective. They propose Reinforced Attention Learning (RAL) to address the difficulty of applying RL techniques that work well for LLMs but tend to struggle more with multimodal LLMs, although I think the idea is more general than that. They argue that this approach is more robust to reward hacking and less prone to diversity collapse. In practice, they add to the standard RL objective an additional term that looks like: E[ A_t · D(p_t^θ || p_t^old) ] Here, A_t is the advantage, while p_t is the attention distribution policy at generation step t, defined from the final attention layer (averaged across heads): the current one (θ) and the one that generated the batch (old). p_t^θ(i) is simply the attention weight from the generated token at position t to a previous token at position i < t, normalized by the sum of attention weights before t. D is the Jensen–Shannon divergence (JSD), a symmetric and bounded divergence defined via the mixture of the two distributions, and less aggressive than Kullback-Leibler divergence (KL). JSD(P || Q) = 1/2 · KL(P || M) + 1/2 · KL(Q || M) With M = 1/2 · (P + Q). What is happening here is that we incentivize attention patterns associated with high advantage and discourage those associated with low advantage. As the authors put it, when A_t < 0 the model is explicitly pushed to explore new reasoning paths. The full objective becomes: L_RL + λ · L_AttnRL. They then introduce an extension that combines RAL with on-policy knowledge distillation and attention distillation. The idea is similar to on-policy distillation: you generate with the student (the current network), compute the teacher outputs on the same generation, and align them using a reverse KL on the logits. In addition, they also align the attention distributions between teacher and student, again using JSD. Unlike RAL, this attention distillation term does not use an advantage, it is pure structural alignment. Takeaway: RL and on-policy distillation do not have to operate only at the token level. Extending the training signal to internal structures like attention can be quite effective, and I expect this direction to be explored much more in the future. Thanks Petru-Daniel (Dan) T. for suggesting the paper!

84

1 Comment

Hallucinations remain one of the most stubborn challenges in large language models... And retrieval-augmented generation (RAG) is still one of the most reliable ways to address them. In A Simple Guide to Retrieval Augmented Generation, Abhinav Kimothi breaks RAG down to first principles, then builds back up through real system designs. The book covers everything from core concepts to production-ready architectures, including agentic variations, with a clear focus on how these ideas actually work in practice. Reviewer Alain AIROM sums it up well, calling the book "excellent" for readers who value hands-on code. Read Alain's full review: https://hubs.la/Q03ZGrgN0 Explore the book: https://hubs.la/Q03ZGJzm0

42

Relay works on a simple principle - populate your semantic abstractions ahead of time, provide a creative brief when you want to write new copy, let the LLM take the first pass as generating semantically tagged copy at the scale your agents need, then iterate from there. Early on, I added a "rules" field to accompany the field that took the creative brief. Users could specify constraints they wanted enforced across all generated content. "Never use exclamation points." "Always mention the product name." "Keep sentences under 15 words." It seemed like a good idea - give users control, let them shape the output before generation. We ultimately removed it. Users simply aren't that good at anticipating what constraints they need, and there’s no reason to expect them to be. They'd leave the rules box empty, generate content, see something they didn't like, and then try to reverse-engineer what rule would have prevented it. Or they'd fill the box with rules based on past problems that didn't apply to the current brief, and the over-constrained output would be worse than if they'd specified nothing at all. We eventually settled on what I think is a better pattern: let users give feedback on outputs rather than rules for inputs. Generate first. Look at what came back. Say "these calls to action are too aggressive" or "I wanted less emphasis on discounts." Let the LLM revise the underlying query to incorporate that feedback, and regenerate. Users are much better at reacting to concrete outputs than at specifying abstract constraints upfront. "I don't like this" is easier than "here are the rules that would prevent things I don't like." The query evolves through contact with outputs, and each round of feedback makes it more precise. The rules box felt clean, but clean isn't always effective.

20

1 Comment

Announcing a new pre-print, "More code, less validation: Risk factors for over-reliance on AI coding tools among scientists" with Alexis Parker, Nasir Eisty and Jeffrey Carver. In a survey of >800 scientists across domains and career stages who program, we find: - ChatGPT is king for programming support, as only a minority of scientific coders are using dedicated developer tools. In general, adoption is highest among student researchers and more junior programmers. - Perceived productivity gains are highest for scientists who are a) less experienced programmers, b) have relatively low use of software development practices like version control, code review, and testing, and c) report accepting more lines of generated code in their interactions with AI tools. - Non-adopters cite concerns about preventing their own learning and ethical objections. Now my editorializing: it is concerning to me the scientists who likely have the least support to get critical feedback about their code (whether through automated tests or review by a lab-mate) and the least experience are most likely to rate themselves as most productive with AI tools. Some of these reported productivity gains could be very real- and at the same time, it could be getting easier than ever to publish scientific code with subtle but serious semantic errors, because of lack of opportunities for feedback beyond runtime errors (which LLM-based tools are pretty good at resolving, anyway). I also wonder if we should be encouraging scientists who are currently programming with ChatGPT towards trying tools that are integrated into developer environments, which might blunt some of the most serious sycophancy-type issues associated with chat and better facilitate "good enough" development practices. Developer tools that use sub-agents also relieve some meta-cognitive burden of tracking context rot, which I suspect is a lot for junior coders. What do you think? Link: https://lnkd.in/gqDVFKMB

58