Overlooked Topics in Large Language Model Research

Can AI models get "Brain Rot"? New research says, Yes! A recent paper on the 'LLM Brain Rot Hypothesis' presents findings that are crucial for anyone involved in AI development. Researchers have discovered that continuous exposure to low-quality web content leads to lasting cognitive decline in large language models (LLMs). The key impacts identified include:- - 17-24% drop in reasoning tasks (ARC-Challenge). - 32% decline in long-context understanding (RULER). - Increased safety risks. - Emergence of negative personality traits (psychopathy, narcissism). What defines "junk data"? Two dimensions are significant:- - Engagement-driven content (short, viral posts). - Low semantic quality (clickbait, conspiracy theories, superficial content). The most concerning finding is that the damage is persistent. Even scaling up instruction tuning and clean data training cannot fully restore baseline capabilities, indicating deep representational drift rather than mere surface-level formatting issues. This research highlights that as we develop autonomous AI systems, data quality transcends being a mere training concern; it becomes a safety issue. We need to implement:- - Routine "cognitive health checks" for deployed models. - Careful curation during continual learning. - A better understanding of how data quality affects agent reliability. The paper emphasizes that data curation for continual pretraining is a training-time safety problem, not just a performance optimization. For those building production AI systems, this research should fundamentally alter our approach to data pipelines and model maintenance. Link to paper: https://lnkd.in/drgjvt8a #AI #MachineLearning #AgenticAI #DataQuality #AIResearch #LLM #AIEthics

LLM: Do We Need a New Acronym for AI? LLMs are the engines behind Generative AI. 🧠A Large Language Model (LLM) is a type of AI trained on huge amounts of text so it can recognise, summarise, translate, predict, and generate language that sounds human. 🗣️LLMs like GPT-4, Claude, and Gemini are driving the AI boom - reshaping how we work, learn, and communicate. But here’s the uncomfortable truth: These models are only as good -or as biased -as the data they are build on. 📚Most LLMs are trained on oceans of human-created content : books, articles, forums, the internet. They absorb our brilliance but also our biases: stereotypes, sexism, racism and deep structural inequalities embedded in language itself. Yes, there’s human oversight to filter out the worst but bias still seeps through with a risk of institutionalising these problems. ⚠️This brings real risks: Whose voices are amplified? Whose stories are erased? Who benefits and who is left behind? So, how we respond or do we just keep making them bigger? 💬At panels and sessions I’ve moderated recently, I’ve asked AI leaders: 👉 Should we build Smaller Language Models (SLMs) — more intentional, domain-specific, and transparent? 👉 Should we design Inclusive Language Models (ILMs) — created from the start with diverse, local, underrepresented voices and perspectives? 👉 Or imagine an All Languages Language Model (ALLM) — one that truly reflects the full richness of global languages and cultures, not just dominant English-speaking tech hubs? The responses have been powerful — and urgently needed. Because AI is not neutral. Inclusion cannot be an afterthought. So here’s a bigger question and a bigger idea. What if we dared to build a GLM -a Global, Gender-Inclusive Language Model? 🌍 A model that centres gender inclusion, diverse identities, and historically excluded communities - by design, not by trying to patching up bias later. 🌎A model built with the all countries not just dominated by the Global North datasets 🌏A model that lifts up underrepresented languages, local knowledge, and cultural context. 🌍A model that flips the script — and asks: Who gets to shape and benefit from the next wave of AI? What do you think? 💡 Idea Would you support building GLMs -Global Inclusive Language Models - to tackle bias and risk, and make AI truly work for everyone? We need new models, new frameworks, and new voices - especially from underrepresented groups too often left out. The future won’t build itself. We need to build it better, together. (📸: Photo taken moderating AI for Good panel)

Like a fortress growing taller but keeping the same cracks, large language models may be expanding without becoming safer. A collaborative study between the UK AI Security Institute, Anthropic, University of Oxford, and the The Alan Turing Institute exposes this unsettling symmetry. The study demonstrates that data poisoning does not dilute with scale. Even as models and datasets grow by orders of magnitude, the absolute number of poisoned samples required to implant a backdoor remains roughly constant. In their experiments, 250 poisoned documents were sufficient to compromise models ranging from 600M to 13B parameters, despite the largest model being trained on nearly twenty times more clean data. This overturns the long-held belief that increasing data volume would naturally “average out” adversarial noise. Instead, larger models appear to be more sample-efficient learners, capable of internalizing both useful and malicious signals with equal precision. For those of us working on trust layers over model training - through Knowledge Graphs, ontology-driven provenance, and dynamic data vetting - this finding reinforces a critical point: robustness is not an emergent property of scale; it must be deliberately engineered. Key implications include: 1) Scaling laws for capability may mirror scaling laws for vulnerability. 2) Fine-tuning or alignment processes cannot reliably erase deeply embedded backdoors; they often only suppress them. 3) Graph-based reasoning layers may become essential for tracing data lineage and identifying subtle poisoning patterns before training. In the pursuit of larger and more capable models, the real challenge is ensuring that every data point shaping them remains interpretable, auditable, and trusted. Scaling safety will demand more than data volume - it will require transparency, traceability, and semantic intelligence across the entire data pipeline. Full length article: https://lnkd.in/gmMNdFgF #AISafety #DataPoisoning #ModelRobustness #BackdoorAttacks #AdversarialAI #AICybersecurity #LLMSecurity #AITrust #AIIntegrity #ResponsibleAI #ScalingLaws #FoundationModels #LargeLanguageModels #ModelAlignment #AIAlignment #ModelScaling #AIResearch #MachineLearningResearch #KnowledgeGraphs #OntologyEngineering #DataLineage #DataProvenance #TrustworthyAI #ExplainableAI #InterpretableAI #SemanticAI #AIEthics #AIGovernance #SafeAI #AITransparency #AIForGood #TechPolicy #DigitalTrust #FutureOfAI #AI #MachineLearning #DeepLearning #GenerativeAI #TechInnovation #EmergingTech

Exciting New Research: Injecting Domain-Specific Knowledge into Large Language Models I just came across a fascinating comprehensive survey on enhancing Large Language Models (LLMs) with domain-specific knowledge. While LLMs like GPT-4 have shown remarkable general capabilities, they often struggle with specialized domains such as healthcare, chemistry, and legal analysis that require deep expertise. The researchers (Song, Yan, Liu, and colleagues) have systematically categorized knowledge injection methods into four key paradigms: 1. Dynamic Knowledge Injection - This approach retrieves information from external knowledge bases in real-time during inference, combining it with the input for enhanced reasoning. It offers flexibility and easy updates without retraining, though it depends heavily on retrieval quality and can slow inference. 2. Static Knowledge Embedding - This method embeds domain knowledge directly into model parameters through fine-tuning. PMC-LLaMA, for instance, extends LLaMA 7B by pretraining on 4.9 million PubMed Central articles. While offering faster inference without retrieval steps, it requires costly updates when knowledge changes. 3. Modular Knowledge Adapters - These introduce small, trainable modules that plug into the base model while keeping original parameters frozen. This parameter-efficient approach preserves general capabilities while adding domain expertise, striking a balance between flexibility and computational efficiency. 4. Prompt Optimization - Rather than retrieving external knowledge, this technique focuses on crafting prompts that guide LLMs to leverage their internal knowledge more effectively. It requires no training but depends on careful prompt engineering. The survey also highlights impressive domain-specific applications across biomedicine, finance, materials science, and human-centered domains. For example, in biomedicine, domain-specific models like PMC-LLaMA-13B significantly outperform general models like LLaMA2-70B by over 10 points on the MedQA dataset, despite having far fewer parameters. Looking ahead, the researchers identify key challenges including maintaining knowledge consistency when integrating multiple sources and enabling cross-domain knowledge transfer between distinct fields with different terminologies and reasoning patterns. This research provides a valuable roadmap for developing more specialized AI systems that combine the broad capabilities of LLMs with the precision and depth required for expert domains. As we continue to advance AI systems, this balance between generality and specialization will be crucial.

Large Language Diffusion Models (LLaDA) Proposes a diffusion-based approach that can match or beat leading autoregressive LLMs in many tasks. If true, this could open a new path for large-scale language modeling beyond autoregression. More on the paper: Questioning autoregressive dominance While almost all large language models (LLMs) use the next-token prediction paradigm, the authors propose that key capabilities (scalability, in-context learning, instruction-following) actually derive from general generative principles rather than strictly from autoregressive modeling. Masked diffusion + Transformers LLaDA is built on a masked diffusion framework that learns by progressively masking tokens and training a Transformer to recover the original text. This yields a non-autoregressive generative model—potentially addressing left-to-right constraints in standard LLMs. Strong scalability Trained on 2.3T tokens (8B parameters), LLaDA performs competitively with top LLaMA-based LLMs across math (GSM8K, MATH), code (HumanEval), and general benchmarks (MMLU). It demonstrates that the diffusion paradigm scales similarly well to autoregressive baselines. Breaks the “reversal curse” LLaDA shows balanced forward/backward reasoning, outperforming GPT-4 and other AR models on reversal tasks (e.g. reversing a poem line). Because diffusion does not enforce left-to-right generation, it is robust at backward completions. Multi-turn dialogue and instruction-following After supervised fine-tuning, LLaDA can carry on multi-turn conversations. It exhibits strong instruction adherence and fluency similar to chat-based AR LLMs—further evidence that advanced LLM traits do not necessarily rely on autoregression. https://lnkd.in/eYp9Hi5y

I recently delved into some intriguing research about the often-overlooked potential of Small Language Models (SLMs). While LLMs usually grab the headlines with their impressive capabilities, studies on SLMs fascinate me because they challenge the “bigger is better��� mindset. They highlight scenarios where smaller, specialized models not only hold their own but actually outperform their larger counterparts. Here are some key insights from the research: 𝟏. 𝐑𝐞𝐚𝐥-𝐓𝐢𝐦𝐞, 𝐏𝐫𝐢𝐯𝐚𝐜𝐲-𝐅𝐨𝐜𝐮𝐬𝐞𝐝 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬: SLMs excel in situations where data privacy and low latency are critical. Imagine mobile apps that need to process personal data locally or customer support bots requiring instant, accurate responses. SLMs can deliver high-quality results without sending sensitive information to the cloud, thus enhancing data security and reducing response times. 𝟐. 𝐒𝐩𝐞𝐜𝐢𝐚𝐥𝐢𝐳𝐞𝐝, 𝐃𝐨𝐦𝐚𝐢𝐧-𝐒𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐓𝐚𝐬𝐤𝐬: In industries like healthcare, finance, and law, accuracy and relevance are paramount. SLMs can be fine-tuned on targeted datasets, often outperforming general LLMs for specific tasks while using a fraction of the computational resources. For example, an SLM trained on medical terminology can provide precise and actionable insights without the overhead of a massive model. 𝟑. 𝐀𝐝𝐯𝐚𝐧𝐜𝐞𝐝 𝐓𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬 𝐟𝐨𝐫 𝐋𝐢𝐠𝐡𝐭𝐰𝐞𝐢𝐠𝐡𝐭 𝐀𝐈: SLMs leverage sophisticated methods to maintain high performance despite their smaller size: • Pruning: Eliminates redundant parameters to streamline the model. • Knowledge Distillation: Transfers essential knowledge from larger models to smaller ones, capturing the “best of both worlds.” • Quantization: Reduces memory usage by lowering the precision of non-critical parameters without sacrificing accuracy. These techniques enable SLMs to run efficiently on edge devices where memory and processing power are limited. Despite these advantages, the industry often defaults to LLMs due to a few prevalent mindsets: • “Bigger is Better” Mentality: There’s a common belief that larger models are inherently superior, even when an SLM could perform just as well or better for specific tasks. • Familiarity Bias: Teams accustomed to working with LLMs may overlook the advanced techniques that make SLMs so effective. • One-Size-Fits-All Approach: The allure of a universal solution often overshadows the benefits of a tailored model. Perhaps it’s time to rethink our approach and adopt a “right model for the right task” mindset. By making AI faster, more accessible, and more resource-efficient, SLMs open doors across industries that previously found LLMs too costly or impractical. What are your thoughts on the role of SLMs in the future of AI? Have you encountered situations where a smaller model outperformed a larger one? I’d love to hear your experiences and insights.

It has always been clear that large language models cannot reason, if you cared to look inside. Not because they are too small or too large, lack data, or need more training, but because there is no understanding to begin with. Reasoning presupposes stable referents, causal structure, and the ability to distinguish belief, inference, and commitment under uncertainty. Language models have none of these. They operate through statistical induction over language, not through comprehension of what symbols refer to or mean. A growing body of recent work now acknowledges this gap and proposes agentic scaffolding as a response: planning loops, tool use, reflection, memory, and multi-agent orchestration. What matters is what these approaches do not claim, and what they therefore do not provide. Agentic LLM systems are not claimed to: understand symbols and ontologies generalize from semantic or causal structure possess grounded referents maintain explicit causal models distinguish truth from usefulness separate belief revision from action optimization perform deduction and abduction over semantic propositions The formalism in this paper quietly reflects these absences. Agentic architectures can certainly behave more effectively. They can search, backtrack, retry, and coordinate across time and tasks. But this is synthetic control, not intelligence or cognition, a control system trying to direct behavior from the outside, while the appearance of intelligence and reasoning is projected onto the system itself. An agentic language model still navigates a maze by colliding with constraints and trying alternative paths, not by understanding the structure of the maze or why a path is a dead end. It makes no difference whether this is done by an elephant or a thousand mice. But this was never a surprise. Without understanding, there is no reasoning, only increasingly performative and elaborate behavior. #AI

Current benchmarks for Large Language Models (LLMs) fail to account for the dynamic, interactive nature fundamental to LLM-based software systems. A new control theoretic approach could revolutionize how we steer these systems towards desired outcomes. 𝐖𝐡𝐲 𝐂𝐨𝐧𝐭𝐫𝐨𝐥 𝐓𝐡𝐞𝐨𝐫𝐲 (𝐂𝐓)? Traditionally, LLM performance is measured using benchmarks like “HellaSwag,” “MMLU,” “TruthfulQA,” or “MATH.” These evaluate how well an LLM answers questions requiring knowledge, reasoning, and mathematical skills. However, these benchmarks overlook the dynamic interactions in LLM-based systems, such as chatbots, where multiple question-answer interactions occur. Users typically steer the LLM in a specific direction, refocusing it when it moves off course. Large context windows in modern LLMs build an internal state over interactions. Understanding and optimizing these dynamic interactions is crucial for developing better LLM systems. This is where control theory (CT) comes in. Originating from engineering, CT studies how to influence a systemtowards a desired state using a “control signal”. CT is widely applicable, from electrical engineering to biological systems and disease control. 𝐊𝐞𝐲 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 𝐂𝐓 𝐚𝐝𝐝𝐫𝐞𝐬𝐬𝐞𝐬 1) When is control possible? 2) What is the cost of control? 3) How computationally intensive is control? These are critical questions for LLM systems. Researchers now presented new results on controlling LLM systems (see comments). 𝐊𝐞𝐲 𝐜𝐨𝐧𝐭𝐫𝐢𝐛𝐮𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐭𝐡𝐞 𝐩𝐚𝐩𝐞𝐫 1) Highlighting Differences from Classical Control Theory: LLM systems are discrete in state and time, unlike systems described by ordinary differential equations. Their state space grows exponentially with the number of tokens, and there is mutual exclusion on control input and generated tokens—at any time, you can either input or receive output from the LLM. 2) Defining Control Theory for LLMs: The focus is on analyzing the “reachable set” of output tokens (see image). 3) Theoretical Results: Upper bounds on the reachable set for self-attention layers show which outputs cannot be reached within the next k tokens given a context. 4) Empirical Results: Demonstrations of lower bounds on the reachable set for popular LLMs reveal that likelihood-based metrics, such as cross-entropy loss, cannot ensure exclusion from the reachable output set, highlighting gaps in our understanding of LLM systems and control theory. The paper concludes with exciting research questions: 1) Can LLMs learn to control each other? 2) Can we find controllable subspaces such as in classical control theory? 3) Can we compose control modules and subsystems into an interpretable, predictable, and effective whole? Exploring these questions may shift our approach from individual models to integrated systems and lead to new ideas beyond LLMs. #genai #llm #machinelearning #ai

Here is an interesting pre-print: Large Language Models Do Not Simulate Human Psychology by Schröder et al.. The idea that large language models such as GPT-4 or the fine-tuned CENTAUR could act as “synthetic participants” in psychological studies is appealing. If they truly behaved like humans, researchers could run experiments faster, cheaper, and without the usual privacy concerns. Some earlier studies even reported near-perfect correlations between LLM moral judgments and human judgments on established test scenarios. This paper takes that optimism to task. The authors argue that LLMs generate text by predicting the next token based on patterns in their training data, not by reasoning about meaning. As long as the task closely matches their training data, the match with human responses can be striking. But once you alter the scenario, by changing just one or two words so that the meaning shifts, human participants change their moral ratings in line with the new context, while LLMs often give nearly identical ratings to both versions. The generalization is happening at the level of wording, not at the level of psychological interpretation. In their study, the authors replicated earlier results with several moral scenarios, then reworded each to alter meaning without changing much of the language. For humans, correlations between ratings of original and reworded items dropped notably, reflecting sensitivity to meaning. For GPT-3.5, GPT-4, Llama-3.1, and CENTAUR, correlations remained extremely high, showing that the models largely ignored the semantic shift. Even CENTAUR, which was trained on millions of psychological responses, behaved almost identically to its base model. The conclusion is clear: while LLMs can be useful tools for piloting experiments, refining materials, or annotating data, they cannot be relied on as stand-alone replacements for human participants. Any psychological research using them must still validate outputs against actual human responses. Read the pre-print here: https://lnkd.in/eGMMqwrA #AIinResearch #LLM #BehavioralScience #ResearchMethods

Our paper got published in JAMA! 🎉 Earlier this year, Suhana Bedi Yutong Liu and I led a paper at Stanford University School of Medicine that highlights critical gaps in evaluating Large Language Models (LLMs) in healthcare. We categorized all 519 relevant studies from 1 Jan 2022 to 19 Feb 2024 into (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty. In doing so, we revealed: - Only 5% used real patient care data in their testing and evaluation. - Key tasks like prescription writing and clinical summarization are underexplored. - The focus on accuracy dominates, while vital aspects like fairness, bias, and toxicity remain largely neglected. - Only 1 study assessed the financial impact of LLMs in healthcare. Why does this matter? - Real patient care data encompasses the complexities of clinical practice, and so a thorough evaluation of LLM performance should mirror clinical performance as closely as possible to truly determine its effectiveness. - There are many high-value administrative tasks in health care that are often labor intensive, requiring manual input and contributing to physician burnout, that are currently chronically understudied. - Only 15.8% of studies conducted any evaluation that delves into how factors such as race and ethnicity, gender, or age affect bias in the model’s output. Future research should place greater emphasis on fairness, bias or toxicity evaluations if we want to stop LLMs from perpetuating bias. - Future evaluations must estimate total implementation costs, including model operation, monitoring, maintenance, and infrastructure adjustments, before reallocating resources from other health care initiatives. The paper calls for standardized evaluation metrics, broader coverage of healthcare applications, and real patient care data to ensure safe and equitable AI integration. This is essential for the responsible adoption of LLMs in healthcare to truly improve patient care. And I am delighted that I get to work on implementing the findings of this research at Coalition for Health AI (CHAI). This paper could not have happened without Nigam Shah's constant support, leadership and guidance, and that of our co-authors Dev Dash Sanmi Koyejo Alison Callahan Jason Fries Michael Wornow Akshay Swaminathan Lisa Lehmann H. Christy Hong, MD MBA Mehr Kashyap Akash Chaurasia Nirav R. Shah Karandeep Singh Troy Tazbaz Arnold Milstein Michael Pfeffer. Thank you also to Nicholas Chedid, MD, MBA Brian Anderson, MD and Justin Norden, MD, MBA, MPhil for your guidance and mentorship. And of course, huge shout out to my co-conspirators Yutong Liu Suhana Bedi - you are the best team. This is the first paper I've ever written, and I'm eternally grateful to you all for showing me how it's done. Full article here: https://lnkd.in/eimh9BNV