Issues With Unstable Knowledge in Large Language Models

Exciting Research Alert: Solving Knowledge Conflicts in Retrieval-augmented LLMs I just came across a fascinating new paper that addresses a critical challenge in retrieval-augmented large language models (LLMs): knowledge conflicts. >> The Problem When LLMs encounter conflicting information between their internal knowledge and retrieved external data, they often struggle to generate reliable responses. This creates uncertainty in decision-making and undermines the reliability of AI systems in real-world applications. >> The Solution: Swin-VIB Researchers from Nankai University, Haihe Lab of ITAI, Meta AI, and InnoPeak Technology, Inc. have developed an innovative framework called Swin-VIB that tackles this problem head-on. Swin-VIB integrates a pipeline of variational information bottleneck models to adaptively augment retrieved information and guide LLM preference during response generation. The approach is grounded in information theory and reveals a key insight: when the difference between conflicting and supplementary information is significant, LLMs confidently settle into stable preferences, but when the distinction is ambiguous, uncertainty increases dramatically. >> How It Works Under the Hood The system operates through a sliding window mechanism that: 1. Extracts attention scores from transformer decoder blocks 2. Compresses these representations through an encoder-decoder bottleneck structure 3. Adaptively accepts or rejects external context based on uncertainty predictions The framework uses a three-step process: - Retrieval: Gathering relevant contexts from external knowledge - Augmentation: Using trained bottlenecks to perform sliding-window inference - Response generation: Producing reliable outputs even with conflicting information >> Impressive Results The results speak for themselves: - Improved single-choice task accuracy by at least 7.54% over competitive baselines - Enhanced performance across single-choice, open-ended QA, and retrieval-augmented generation (RAG) tasks - Significantly reduced output uncertainty in LLMs - Maintained high-quality response generation without compromising authenticity or coherence This research represents a significant step forward in making retrieval-augmented LLMs more reliable in the wild, especially when dealing with misinformation, biases, or outdated knowledge. What are your thoughts on this approach to handling knowledge conflicts in AI systems? Have you encountered similar challenges in your work with LLMs?

Mixture-of-Experts (MoE) LLMs are more prone to training instability than standard LLMs. Here’s why this is the case and how we can fix it… Where do instabilities come from? There are two main issues that occur when training an MoE: 1. Routing collapse: the model converges to using the same expert(s) over and over. 2. Numerical instability: the MoE experiences round-off errors, especially in the router. These issues lead to training instability, meaning that the model’s loss may simply diverge (i.e., go up instead of down) during the training process. Avoiding routing collapse: We need to add auxiliary losses to our training objective that encourage the model to use experts uniformly. The most common auxiliary loss for MoEs is the load balancing auxiliary loss [1], which is minimized when the MoE i) assigns probability uniformly to experts and ii) routes an equal number of tokens to each expert within a batch. Avoiding numerical instability: The biggest source of numerical instability occurs in the MoE’s router because the router includes an (exponential) softmax function. To avoid numerical instabilities in this layer, we can add an auxiliary loss that encourages the values going into the softmax function to not be too large–this is called the router z-loss [2]. Although many LLMs are trained in lower (bfloat16) precision, we should avoid using low precision within the router. Mixed / low precision training greatly improves training efficiency, but it can also make round-off errors more frequent within the router! Weight initialization: Traditionally, we made the training of large, deep neural networks more stable by discovering better weight initialization (e.g., He or Glorot init) and normalization (e.g., batch normalization) techniques. Similarly, we can improve MoE training stability by using a weight initialization strategy that’s more tailored to MoEs. As proposed in [1], we can sample from a truncated normal distribution with a mean of zero (µ = 0) and standard deviation given by σ = SQRT(s/n), where s (0.1 by default) is a scale hyperparameter and n is the size of the input to the layer being initialized. Putting everything together: I’ve tried out each of these techniques within nanoMoE, a simple and functional MoE pretraining implementation that I recently released. We can see that each of these tricks improves the MoE’s training stability. When we use them all together, nanoMoE is able to fully complete pretraining without having any instabilities!

Tackling Hallucination in LLMs: Mitigation & Evaluation Strategies As Large Language Models (LLMs) redefine how we interact with AI, one critical challenge is hallucination—when models generate false or misleading responses. This issue affects the reliability of LLMs, particularly in high-stakes applications like healthcare, legal, and education. To ensure trustworthiness, it’s essential to adopt robust strategies for mitigating and evaluating hallucination. The workflow outlined above presents a structured approach to addressing this challenge: 1️⃣ Hallucination QA Set Generation Starting with a raw corpus, we process knowledge bases and apply weighted sampling to create diverse, high-quality datasets. This includes generating baseline questions, multi-context queries, and complex reasoning tasks, ensuring a comprehensive evaluation framework. Rigorous filtering and quality checks ensure datasets are robust and aligned with real-world complexities. 2️⃣ Hallucination Benchmarking By pre-processing datasets, answers are categorized as correct or hallucinated, providing a benchmark for model performance. This phase involves tools like classification models and text generation to assess reliability under various conditions. 3️⃣ Hallucination Mitigation Strategies In-Context Learning: Enhancing output reliability by incorporating examples directly in the prompt. Retrieval-Augmented Generation: Supplementing model responses with real-time data retrieval. Parameter-Efficient Fine-Tuning: Fine-tuning targeted parts of the model for specific tasks. By implementing these strategies, we can significantly reduce hallucination risks, ensuring LLMs deliver accurate and context-aware responses across diverse applications. 💡 What strategies do you employ to minimize hallucination in AI systems? Let’s discuss and learn together in the comments!

LLMs are powerful— but can you trust them? Every AI founder knows the real problem isn’t just building large language models…it’s making sure they don’t lie with confidence. Hallucinations are the Achilles' heel of modern LLMs. They generate plausible-sounding but completely fabricated information, which is a problem when you’re deploying AI in medicine, finance, security, or any high-stakes field. And the worst part? The more advanced these models get, the better they get at making up convincing nonsense. At Future AGI, we tackle this head-on. 1️⃣ Garbage in, garbage out. Bad training data? Not on our watch. We leverage high-quality synthetic data and annotated production data— a gold mine for training more reliable models. 2️⃣ Instead of relying on generic uncertainty estimation, we build business-specific evaluation systems to fine-tune prompt behavior in real-world applications. 3️⃣ External validation is non-negotiable. Knowledge bases, retrieval-augmented generation (RAG), and multi-modal evaluations help ground responses in reality. 4️⃣ We run scientific experimentation on prompt outputs to understand what works, what doesn’t, and how to systematically reduce hallucinations without unnecessary compute overhead. 5️⃣ We don’t just tweak prompts— we auto-tune them. Our "Optimize" feature dynamically adjusts prompts for better accuracy, while the "Experiment" feature lets users test variations across different LLMs and datasets, selecting the best-performing one. How is your team handling hallucinations in AI? My comments are open. 

To Believe or Not to Believe Your LLM We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider both epistemic and aleatoric uncertainties, where the former comes from the lack of knowledge about the ground truth (such as about facts or the language), and the latter comes from irreducible randomness (such as multiple possible answers). In particular, we derive an information-theoretic metric that allows to reliably detect when only epistemic uncertainty is large, in which case the output of the model is unreliable. This condition can be computed based solely on the output of the model obtained simply by some special iterative prompting based on the previous responses. Such quantification, for instance, allows to detect hallucinations (cases when epistemic uncertainty is high) in both single- and multi-answer responses. This is in contrast to many standard uncertainty quantification strategies (such as thresholding the log-likelihood of a response) where hallucinations in the multi-answer case cannot be detected. We conduct a series of experiments which demonstrate the advantage of our formulation. Further, our investigations shed some light on how the probabilities assigned to a given output by an LLM can be amplified by iterative prompting, which might be of independent interest.

The last week was full of learning and discussions with the AI research community at NeurIPS, where Prof Shivani Shukla and I presented two papers that challenge how we think about deploying Gen and agentic AI systems in a secure and safe manner. After months of rigorous research and experimentation, our research group was delighted to have shared findings that bridge critical gaps in our understanding of LLM behavior and human-AI collaboration for the following two papers/posters:- 1. Security Knowledge Dilution in Large Language Models Paper:- https://lnkd.in/dPkPtCRD for workshop Deep Learning for Code in Agentic Era (https://lnkd.in/eMpGGAwg) Our controlled study of 400 experiments revealed a striking finding:- LLMs experience a 47% degradation in security expertise when exposed to large volumes of irrelevant context. This has profound implications for deploying AI systems in security-critical environments where context windows are flooded with operational data. 2. A Stochastic Differential Equation Framework for Multi-Objective LLM Interactions Paper:- https://lnkd.in/dQEPpGmV for workshop DynaFront : Dynamics at the Frontiers of Optimization, Sampling, and Games (https://lnkd.in/eAJK52Bb) Presenting our mathematical framework for understanding how language models navigate competing objectives in real-time interactions, essential for building robust agentic AI systems that can balance multiple constraints simultaneously. These aren't just academic exercises. As we deploy increasingly autonomous AI agents in enterprise environments, understanding how context affects domain expertise and how models reconcile competing objectives becomes mission-critical for responsible AI deployment. The conversations at NeurIPS pushed us to think harder about building systems that are not just powerful, but reliably safe and effective at scale. Grateful to everyone who engaged with our work and challenged our assumptions, that's where the real breakthroughs happen. For those building agentic AI solutions:- How are you addressing context management and multi-objective optimization in your deployments? These challenges are only growing as we scale. #NeurIPS2025 #AIResearch #AgenticAI #AIGovernance #MachineLearning #ResponsibleAI

Like a fortress growing taller but keeping the same cracks, large language models may be expanding without becoming safer. A collaborative study between the UK AI Security Institute, Anthropic, University of Oxford, and the The Alan Turing Institute exposes this unsettling symmetry. The study demonstrates that data poisoning does not dilute with scale. Even as models and datasets grow by orders of magnitude, the absolute number of poisoned samples required to implant a backdoor remains roughly constant. In their experiments, 250 poisoned documents were sufficient to compromise models ranging from 600M to 13B parameters, despite the largest model being trained on nearly twenty times more clean data. This overturns the long-held belief that increasing data volume would naturally “average out” adversarial noise. Instead, larger models appear to be more sample-efficient learners, capable of internalizing both useful and malicious signals with equal precision. For those of us working on trust layers over model training - through Knowledge Graphs, ontology-driven provenance, and dynamic data vetting - this finding reinforces a critical point: robustness is not an emergent property of scale; it must be deliberately engineered. Key implications include: 1) Scaling laws for capability may mirror scaling laws for vulnerability. 2) Fine-tuning or alignment processes cannot reliably erase deeply embedded backdoors; they often only suppress them. 3) Graph-based reasoning layers may become essential for tracing data lineage and identifying subtle poisoning patterns before training. In the pursuit of larger and more capable models, the real challenge is ensuring that every data point shaping them remains interpretable, auditable, and trusted. Scaling safety will demand more than data volume - it will require transparency, traceability, and semantic intelligence across the entire data pipeline. Full length article: https://lnkd.in/gmMNdFgF #AISafety #DataPoisoning #ModelRobustness #BackdoorAttacks #AdversarialAI #AICybersecurity #LLMSecurity #AITrust #AIIntegrity #ResponsibleAI #ScalingLaws #FoundationModels #LargeLanguageModels #ModelAlignment #AIAlignment #ModelScaling #AIResearch #MachineLearningResearch #KnowledgeGraphs #OntologyEngineering #DataLineage #DataProvenance #TrustworthyAI #ExplainableAI #InterpretableAI #SemanticAI #AIEthics #AIGovernance #SafeAI #AITransparency #AIForGood #TechPolicy #DigitalTrust #FutureOfAI #AI #MachineLearning #DeepLearning #GenerativeAI #TechInnovation #EmergingTech

RAG (retrieval-augmented generation) has been recognized as a method to reduce hallucinations in LLMs, but is it really as reliable as many of us think it is? The timely research "How faithful are RAG models? Quantifying the tug-of-war between RAG and LLMs' internal prior" resonated with our struggles when LLMs don't always follow external knowledge in RAG systems, even when ground truth (from a knowledge graph) is provided. One interesting takeaway is that, like humans, different language models have varying degrees of "stubbornness." Some models are more likely to fall back on their internal knowledge when external information contrasts with what they have been trained on. Essentially, these models struggle to adjust to new values or patterns that differ significantly from their existing "belief system." We also found that a knowledge graph-based method, the entity linker, can enhance the correctness of answers by filtering out erroneous information that doesn't match the ground truth in knowledge graphs. Check out how the entity linker solved the funny hallucination where "PayPal" was recognized as a co-founder of SpaceX in the video at 4:21, with RAG and an entity linker outperforming the one without it. Testing our "optimized" DSPy + KG pipeline and seeing LLMs' unpredictable reasoning patterns made me realize we should probably assess the nature of our tasks more carefully before fully relying on LLMs to automate them?