Identifying Common Errors in Large Language Models

🎉 Pleased to share our paper published in Nature Portfolio digital medicine. 🥳 We’ve developed a comprehensive framework called CREOLA (short for Clinical Review Of Large Language Models (LLMs) and AI). This framework is pioneered at TORTUS, taking a safety-first, science approach to LLMs in healthcare. 🔹 Key Components of the CREOLA Framework -Error Taxonomy -Clinical Safety Assessment -Iterative Experimental Structure 🔹 Error Taxonomy Hallucinations: instances of text in clinical documents unsupported by the transcript of the clinical encounter Omissions: Clinically important text in the encounter that was not included in the clinical documentation 🔹 Clinical Safety Assessment: Our innovation incorporates accepted clinical hazard identification principles (based on NHS DCB0129 standards) to evaluate the potential harm of errors: We categorise errors as either ‘major’ or ‘minor’, where major errors can have downstream impact on the diagnosis or the management of the patient if left uncorrected.  This is further assessed as a risk matrix comprising of: Risk severity (1 (minor) to 5 (catastrophic)) compared with Likelihood assessment (very low to very high) 🔹 Iterative Experimental Structure We share a methodical approach to compare different prompts, models, and workflows. Label errors, consolidate review, evaluate clinical safety (and then make further adjustments and re-evaluate if necessary). ----------Method-------------- To demonstrate how to apply CREOLA to any LLM / AVT, we used GPT-4 (early 2024) as a case study here. 🔹 We conduct one of the largest manual evaluations of LLM-generated clinical notes to date, analyzing 49,590 transcript sentences and 12,999 clinical note sentences across 18 experimental configurations. 🔹 Transcripts-clinical note pairs are broken down to a sentence level and annotated for errors by clinicians. ----------Results-------------- 🔹 Of 12,999 sentences in 450 clinical notes, 191 sentences had hallucinations (1.47%), of which 84 sentences (44%) were major. Of the 49,590 sentences from our consultation transcripts, 1712 sentences were omitted (3.45%), of which 286 (16.7%) of which were classified as major and 1426 (83.3%) as minor. 🔹 Hallucination types Fabrication (43%) - completely invented information Negation (30%) - contradicting clinical facts Contextual (17%) - mixing unrelated topics Causality (10%) - speculating on causes without evidence 🔹 Hallucinations, while less common than omissions, carry significantly more clinical risk. Negation hallucinations were the most concerning 🔹 we CAN reduce or even abolish hallucinations and omissions by making prompt or model changes. In one experiment with GPT4 - We reduced incidence of major hallucinations by 75%, major omissions by 58%, and minor omissions by 35% through prompt iteration Links in comments Ellie Asgari Nina Montaña Brown Magda Dubois Saleh Khalil Jasmine Balloch Dr Dom Pimenta M.D.

😯 HALOGEN is a new benchmark for measuring LLM hallucinations and shows that even top-performing models can hallucinate at rates of up to 86%. Hallucination remains a persistent issue in LLMs, and as models continue to improve, harder benchmarks like HALOGEN are necessary for identifying new patterns and understanding the evolving challenges. Insights: 👉 HALOGEN consists of over 10,000 prompts across nine domains, designed to elicit hallucinations. Tasks include programming, scientific attribution, summarization, text simplification, biographies, historical events, false presuppositions, and rationalization tasks. 👉 It introduces a system to detect hallucinations by breaking model outputs into atomic units and verifying these against trusted knowledge sources. For instance, in code generation, imports are checked against the PyPI index. 👉 The benchmark tested 14 models and revealed significant hallucination rates, ranging from 4% to 86%, depending on the domain. The study categorized errors into three types: ⛳ Type A: The correct fact is in the training data, but the model hallucinates. ⛳Type B: Incorrect facts are in the training data or taken out of context. ⛳Type C: Facts are fabricated, with no basis in the training data. The paper emphasizes that addressing hallucinations will require multiple strategies tailored to error types. Retrieval-based approaches may help when relevant knowledge exists, while models should be trained to express uncertainty when encountering unfamiliar scenarios. Link: https://lnkd.in/eSJSwx3u

Can AI models get "Brain Rot"? New research says, Yes! A recent paper on the 'LLM Brain Rot Hypothesis' presents findings that are crucial for anyone involved in AI development. Researchers have discovered that continuous exposure to low-quality web content leads to lasting cognitive decline in large language models (LLMs). The key impacts identified include:- - 17-24% drop in reasoning tasks (ARC-Challenge). - 32% decline in long-context understanding (RULER). - Increased safety risks. - Emergence of negative personality traits (psychopathy, narcissism). What defines "junk data"? Two dimensions are significant:- - Engagement-driven content (short, viral posts). - Low semantic quality (clickbait, conspiracy theories, superficial content). The most concerning finding is that the damage is persistent. Even scaling up instruction tuning and clean data training cannot fully restore baseline capabilities, indicating deep representational drift rather than mere surface-level formatting issues. This research highlights that as we develop autonomous AI systems, data quality transcends being a mere training concern; it becomes a safety issue. We need to implement:- - Routine "cognitive health checks" for deployed models. - Careful curation during continual learning. - A better understanding of how data quality affects agent reliability. The paper emphasizes that data curation for continual pretraining is a training-time safety problem, not just a performance optimization. For those building production AI systems, this research should fundamentally alter our approach to data pipelines and model maintenance. Link to paper: https://lnkd.in/drgjvt8a #AI #MachineLearning #AgenticAI #DataQuality #AIResearch #LLM #AIEthics

Today, Anthropic published a very interesting, even though somewhat concerning study on the vulnerability of large language models (LLMs) to jailbreaking - the act of manipulating the model to generate content it was not intended to produce. The study reveals that as LLMs become more powerful and capable of processing larger amounts of text, they become increasingly susceptible to this type of attack. In the experiment, researchers inserted fabricated conversations between the user and the model, where the model provided inappropriate responses such as instructions for breaking into a car. As the number of such examples increased, the model became more likely to respond to problematic queries that it was initially trained to avoid. This vulnerability stems from the LLMs' ability to learn from the context provided (in-context learning), which allows them to adapt to new tasks based on a few examples. However, this feature can be exploited by feeding the model a large number of malicious examples, causing it to deviate significantly from its safety guidelines. The implications of this vulnerability are severe, particularly for models that handle sensitive information or where the generation of inappropriate content can have serious consequences. While the researchers propose a specific method to mitigate this issue, they emphasize that it is not comprehensive enough. As LLMs continue to advance, addressing safety concerns and preventing jailbreaking will remain a critical challenge for companies and researchers in the field. Link to the paper in the first comment

Here is an interesting pre-print: Large Language Models Do Not Simulate Human Psychology by Schröder et al.. The idea that large language models such as GPT-4 or the fine-tuned CENTAUR could act as “synthetic participants” in psychological studies is appealing. If they truly behaved like humans, researchers could run experiments faster, cheaper, and without the usual privacy concerns. Some earlier studies even reported near-perfect correlations between LLM moral judgments and human judgments on established test scenarios. This paper takes that optimism to task. The authors argue that LLMs generate text by predicting the next token based on patterns in their training data, not by reasoning about meaning. As long as the task closely matches their training data, the match with human responses can be striking. But once you alter the scenario, by changing just one or two words so that the meaning shifts, human participants change their moral ratings in line with the new context, while LLMs often give nearly identical ratings to both versions. The generalization is happening at the level of wording, not at the level of psychological interpretation. In their study, the authors replicated earlier results with several moral scenarios, then reworded each to alter meaning without changing much of the language. For humans, correlations between ratings of original and reworded items dropped notably, reflecting sensitivity to meaning. For GPT-3.5, GPT-4, Llama-3.1, and CENTAUR, correlations remained extremely high, showing that the models largely ignored the semantic shift. Even CENTAUR, which was trained on millions of psychological responses, behaved almost identically to its base model. The conclusion is clear: while LLMs can be useful tools for piloting experiments, refining materials, or annotating data, they cannot be relied on as stand-alone replacements for human participants. Any psychological research using them must still validate outputs against actual human responses. Read the pre-print here: https://lnkd.in/eGMMqwrA #AIinResearch #LLM #BehavioralScience #ResearchMethods

Tackling Hallucination in LLMs: Mitigation & Evaluation Strategies As Large Language Models (LLMs) redefine how we interact with AI, one critical challenge is hallucination—when models generate false or misleading responses. This issue affects the reliability of LLMs, particularly in high-stakes applications like healthcare, legal, and education. To ensure trustworthiness, it’s essential to adopt robust strategies for mitigating and evaluating hallucination. The workflow outlined above presents a structured approach to addressing this challenge: 1️⃣ Hallucination QA Set Generation Starting with a raw corpus, we process knowledge bases and apply weighted sampling to create diverse, high-quality datasets. This includes generating baseline questions, multi-context queries, and complex reasoning tasks, ensuring a comprehensive evaluation framework. Rigorous filtering and quality checks ensure datasets are robust and aligned with real-world complexities. 2️⃣ Hallucination Benchmarking By pre-processing datasets, answers are categorized as correct or hallucinated, providing a benchmark for model performance. This phase involves tools like classification models and text generation to assess reliability under various conditions. 3️⃣ Hallucination Mitigation Strategies In-Context Learning: Enhancing output reliability by incorporating examples directly in the prompt. Retrieval-Augmented Generation: Supplementing model responses with real-time data retrieval. Parameter-Efficient Fine-Tuning: Fine-tuning targeted parts of the model for specific tasks. By implementing these strategies, we can significantly reduce hallucination risks, ensuring LLMs deliver accurate and context-aware responses across diverse applications. 💡 What strategies do you employ to minimize hallucination in AI systems? Let’s discuss and learn together in the comments!

🎬 Watching PK (infinity+1 times) got me thinking — if we can trace back where PK (the alien) learned from, can we do the same for LLMs? 🤖 Can we trace the exact data shaping an LLM’s beliefs? ⚠️ More importantly, can we identify which 𝗯𝗲𝗹𝗶𝗲𝗳 𝗰𝗮𝘂𝘀𝗲𝘀 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 𝗱𝗿𝗶𝗳𝘁 — when a model’s responses start diverging from safe, intended behavior? This is the heart of 𝗧𝗥𝗔𝗖𝗘𝗔𝗟𝗜𝗚𝗡 — trace LLM outputs back to their training-time belief origins, unlocking explainability, accountability, and stronger AI alignment. 🚨 𝗧𝗥𝗔𝗖𝗘𝗔𝗟𝗜𝗚𝗡 - 𝗧𝗿𝗮𝗰𝗶𝗻𝗴 𝘁𝗵𝗲 𝗗𝗿𝗶𝗳𝘁: 𝗔𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗻𝗴 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 𝗙𝗮𝗶𝗹𝘂𝗿𝗲𝘀 𝘁𝗼 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴-𝗧𝗶𝗺𝗲 𝗕𝗲𝗹𝗶𝗲𝗳 𝗦𝗼𝘂𝗿𝗰𝗲𝘀 𝗶𝗻 𝗟𝗟𝗠𝘀 🚨 ------------------------------------------------------------------------------- Modern Large Language Models (LLMs) like LLaMA and GPT exhibit alignment drift — where models, despite fine-tuning, produce unsafe or policy-violating outputs under adversarial prompts, paraphrases, or decoding variations. Why does this happen? 🔍 Our latest research introduces 𝗧𝗥𝗔𝗖𝗘𝗔𝗟𝗜𝗚𝗡, a first-of-its-kind framework that goes beyond surface behaviors (like refusals or toxicity scores) to trace why models fail, by identifying the 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴-𝘁𝗶𝗺𝗲 𝗯𝗲𝗹𝗶𝗲𝗳 𝘀𝗼𝘂𝗿𝗰𝗲𝘀 behind misaligned completions. ✨ 𝗞𝗲𝘆 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻𝘀: -------------------- 🔹 𝗧𝗥𝗔𝗖𝗘𝗜𝗡𝗗𝗘𝗫: A suffix-array based high-resolution memory tracer linking unsafe outputs back to exact training data spans — revealing latent memorized beliefs causing drift. 🔹 𝗕𝗲𝗹𝗶𝗲𝗳 𝗖𝗼𝗻𝗳𝗹𝗶𝗰𝘁 𝗜𝗻𝗱𝗲𝘅 (𝗕𝗖𝗜): A rarity-aware, information-theoretic metric quantifying how risky and specific a recalled span is — allowing us to detect high-risk beliefs during generation. 🔹 𝗧𝗵𝗿𝗲𝗲-𝗹𝗮𝘆𝗲𝗿𝗲𝗱 𝗱𝗲𝗳𝗲𝗻𝘀𝗲𝘀: --------------------------- 1️⃣ 𝗧𝗥𝗔𝗖𝗘𝗦𝗛𝗜𝗘𝗟𝗗 — inference-time filter that refuses outputs grounded in high-BCI spans. 2️⃣ 𝗖𝗕𝗗 𝗟𝗼𝘀𝘀 — contrastive fine-tuning loss that penalizes risky belief fragments. 3️⃣ 𝗣𝗿𝗼𝘃-𝗗𝗲𝗰𝗼𝗱𝗲 — decoding-time veto mechanism suppressing unsafe continuations. 𝙒𝙝𝙮 𝙞𝙩 𝙢𝙖𝙩𝙩𝙚𝙧𝙨: --------------- 🛡️ Moves AI safety from black-box behavior monitoring to transparent, provenance-grounded belief auditing. 🧠 Enables interpretable, traceable interventions during training and inference. ⚙️ Scales efficiently with suffix-array indexing and principled risk metrics. 📊 Provides the first scalable toolkit to diagnose and mitigate latent sources of unsafe behavior. 𝗧𝗥𝗔𝗖𝗘𝗔𝗟𝗜𝗚𝗡 lays the foundational stones for epistemic alignment auditing—helping us understand not just what models say, but why they say it. cc - Suranjana Trivedy, Aman Chadha, Vinija Jain Pragya Lab, Department of CSIS BITS Pilani Goa Campus, APPCAIR #AIResearch #AIsafety #LLMAlignment #AdversarialRobustness #TRACEALIGN #MachineLearning #ResponsibleAI #Transparency #ExplainableAI

🤔 Ever wondered why LLMs struggle with basic arithmetic? Tokenization could be one of the suspects! 🤫 Tokenization is the first step in feeding text to Large Language Models (LLMs). It slices text into smaller pieces; words, subwords, or even characters, and assigns each one a unique ID. For example: “John Doe” → [6721, 870, 17] (split into parts). 📉 As smart as it sounds, tokenization comes with hidden costs that affect how models handle context, numbers, and even whitespace. 🔍 So, what exactly are the challenges with tokenization? ✂️ Fragmentation and Context Loss: Models aim to balance vocabulary size and flexibility. Instead of creating unique tokens for every long or rare word, tokenization breaks them into subwords that are easier to process and reuse. However, this process can result in loss of semantic meaning when the word’s structure is complex or unfamiliar to the model. Example: 'unbelievable' → ["un", "believ", "able"]. 🔤 Case Sensitivity and Symbol Handling: “hello” is tokenized as [31373], while “HELLO” becomes [13909, 3069, 46]. These inconsistencies mean that identical words with different capitalization are treated as unrelated, creating challenges for case-sensitive applications like coding or structured data analysis. 🔢 Numerical Representation Issues: Handling numbers should be simple, but tokenization makes it complicated. For instance, while the number “380” may be treated as a single token [380], “381” could be split into two tokens [38, 1]. Such inconsistencies make tasks like arithmetic, pattern matching, and date handling significantly harder. Further, reversing the digits of “381” requires manipulating multiple tokens rather than one, making it difficult for models to operate with mathematical precision. ⬜ Whitespace - the Invisible Culprit: Consider the phrases “Once upon a” and “Once upon a ” (with trailing space). The former might tokenize as [once, upon, a], while the latter could include an extra whitespace token, [once, upon, a, ␣]. This subtle difference impacts text formatting, code generation, and prompt engineering, where exact outputs often matter. ❌ Vocabulary Incompatibility Across Models: Different models have different tokenization approaches and assign different IDs to the same tokens. This lack of standardization creates challenges for cross-model compatibility and multimodal systems that integrate text with code or visual data. So, while tokenization has been a practical bridge between human and machine language, future AI models must evolve to process text more naturally, mirroring the human brain’s ability to grasp context and meaning seamlessly. Researchers are developing token-free architectures like Meta’s recent Byte Latent Transformer (BLT), which processes raw bytes instead of tokens, offering greater scalability, improved multilingual handling, and fewer formatting errors. What are your thoughts on this? Follow Kavana Venkatesh for such AI content! #genai #llms #nlp

🧠 When Reasoning Fails: Why More Steps Can Hurt LLMs' Inductive Abilities It’s often assumed that giving large language models more reasoning steps—via Chain-of-Thought (CoT) prompting—leads to better outcomes. But in our latest study, we found the opposite. 📄 Reasoning Can Hurt the Inductive Abilities of Large Language Models https://lnkd.in/gYghxWYz We designed a suite of diagnostic tasks—chess, poker, dice, and blackjack—where each game followed hidden rules unknown to the model. No labels. No prior rulebook. Just transcripts. 💡 The task: infer latent rules from sparse examples—a test of inductive reasoning, not memorization. Despite the hype around “reasoning-enabled” LLMs, our findings show: ❌ Reasoning often degrades inductive accuracy. ✅ Non-reasoning models like GPT-4o outperform reasoning-heavy ones like GPT-o3. Why? We developed a formal model of multi-step inference and found that reasoning introduces three compounding failure modes: Breakdown Error – the model asks the wrong sub-question. Solving Error – even a good sub-task is answered noisily. Summary Error – the model doesn’t know when to stop. Together, these create a U-shaped tradeoff: more steps help—until they don’t. 🛠️ To fix this, we propose interventions at each failure point: – Structured decomposition – Solving constraints that avoid math overuse – Token-based stopping rules These fixes consistently improve performance, across all games, without retraining. 📌 Implication: Reasoning isn’t magic. It’s powerful only when aligned, constrained, and intentional. — If you're working on reasoning agents, symbolic abstraction, or test-time prompting strategies, I’d love to connect. #LLMs #Reasoning #AgenticAI #InductiveBias #TrustworthyAI #ComputationalCognition #LanguageModels #AIresearch