Understanding LLM Knowledge Retention

LLM agents today suffer from a fundamental knowledge retention problem: every task is treated as a blank slate, with no mechanism to accumulate and reuse expertise from past executions. The agent that successfully navigated a complex hotel booking workflow yesterday has zero memory of that experience when faced with a similar task today. This inability to learn from operational history means repeated failures, redundant reasoning steps, and an inability to handle procedural coordination at scale. Existing approaches like ExpeL, AutoGuide, and AutoManual attempt to address this by extracting experience as flattened textual knowledge from execution traces. While useful for simple heuristics, these text-based representations fundamentally cannot capture the procedural logic of complex subtasks that involve sequential coordination, conditional branching, and state tracking. They also lack any maintenance mechanism, meaning the experience repository degrades over time as redundant and obsolete patterns accumulate, bloating the context window and degrading retrieval quality. AutoRefine (https://lnkd.in/e82wv_PR) introduces a dual-form experience pattern framework that goes beyond text. For complex procedural subtasks, it automatically extracts specialized subagents with independent reasoning and memory, effectively encapsulating multi-step coordination logic as reusable autonomous modules. For simpler strategic knowledge, it extracts skill patterns as guidelines or code snippets. A continuous maintenance mechanism scores patterns on effectiveness, frequency, and precision, then prunes the bottom 20% and merges redundant entries to keep the repository compact. Take a read and keep a lookout for the implementation for a real world scenario soon!

Fine-tuning for making expert, domain-specific models? Not so fast! I often get asked whether companies should fine-tune LLMs to internalize the knowledge required for their particular use case or domain. The answer I give is probably not…. There is research suggesting that large language models struggle to acquire new factual knowledge through fine-tuning. Novel knowledge is learned more slowly than knowledge consistent with what the model already knows. This same research also showed that when knowledge is eventually learned from novel examples, there is a linear increase in the model's tendency to hallucinate. Ouch! So what can you do? What should you do? RAG is one approach, but that comes with complexity and its own challenges: RAG pipelines are more complex, with larger storage costs, higher memory and compute requirements (due to longer contexts demanded by the additional context) and higher latency, due to the need to query an external index. In the long term, storing knowledge natively in the model's parameters may also provide generalization advantages, as the model can relate different pieces of knowledge in its parameters. This is particularly apparent for complex or indirect queries, where simple retrieval augmentation may fall short. A very exciting recent paper from Meta introduced a new approach called Active Reading. This approach leverages synthetic data to have LLMs generate a range of diverse training data based on a closed body of knowledge. By having the LLMs read and restructure the data in many and varied ways and training on that enlarged, restructured corpus, you can significantly improve the model's retention of the contained facts. Active Reading applies the same principles observed in human studying, allowing the model itself to propose multiple study strategies — e.g., paraphrasing, knowledge linking, active recall, etc. — and instantiates these different strategies on a document-by-document basis. This process results in a highly diverse and contextually grounded signal which can then be trained on. The authors demonstrate huge gains vs. vanilla fine-tuning: +313% and +160% (relative improvement over vanilla fine-tuning) on SimpleQA and FinanceBench respectively. They also trained a SOTA 8B model for factual QA, demonstrating the utility of the technique at pre-training scale (1T tokens). It should be noted that the Active Reading paper focuses on knowledge acquisition; that traditional fine tuning can still be useful for instilling style, format, reasoning patterns, or other behaviors. Learning Facts at Scale with Active Reading https://lnkd.in/e7FCAq-3 Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? https://lnkd.in/e_REAVZB

The interview is for an AI Agentic Systems Engineer role at Anthropic. The question lands: Interviewer: "We're building autonomous agents for complex, multi-step reasoning over extended periods. How do you tackle the long-term memory problem beyond just increasing context window size?" This is how you answer. You know the context window is a temporary fix. For true long-term memory, an external, dynamic, and structured memory system is key...describe a blend of episodic and semantic memory. You: "An LLM's context window isn't enough. We need an external memory system, combining episodic and semantic approaches." Interviewer: "Break that down. What are these memory types and how do they interact?" You: "Think of it like human memory:" 1. Episodic Memory (Experiences): - Purpose: Stores specific past events (actions, observations, outcomes) in chronological order. The 'what happened when.' - Implementation: A log of structured tuples (timestamp, action, observation). Can be a simple database or a vector store for semantic search over experiences. 2. Semantic Memory (Knowledge & Skills): - Purpose: Stores generalized knowledge, learned facts, successful strategies. The 'what I know' and 'how to do things.' - Implementation: Primarily a vector database for facts, perhaps a knowledge graph for relationships, and a 'skill library' of reusable sub-routines. Interviewer: "How does the agent decide what to store and retrieve?" You: "The LLM orchestrates this, but with explicit processes:" 1. Encoding: The LLM summarizes observations/actions into concise memory chunks. A 'reflection' module can periodically synthesize new semantic knowledge from episodic memories. 2. Retrieval (Recall): - The LLM generates a memory query based on its current goal. - This query searches the vector database (semantic) or structured log (episodic). - The LLM then re-ranks retrieved memories for relevance before integrating them into its prompt. 3. Forgetting/Consolidation: Important to manage growth. Strategies include: - Recency bias, importance weighting (LLM-assigned scores). - Consolidating old episodic memories into new semantic entries to reduce redundancy. Interviewer: "What's the biggest challenge here for a model like Claude?" You: "Ensuring the LLM effectively uses retrieved memory, rather than being overwhelmed. It must learn when to consult memory and what type to query. This involves fine-tuning the LLM on meta-cognitive tasks - teaching it to manage its own knowledge." This shows that you're ready to build truly intelligent, persistent agents. #AI #AgenticAI #LLMs #MemorySystems

Colleagues, given all the excitement about LLMs, I thought you might be interested in my recent paper on whether LLMs "understand" their knowledge. Large Language Models (LLMs) are often criticized for lacking true "understanding" and the ability to "reason" with their knowledge, being seen merely as autocomplete systems. I believe that this assessment might be missing a nuanced insight. I suggest that LLMs do develop a kind of empirical "understanding" that is "geometry"-like, which seems adequate for a range of applications in NLP, computer vision, coding assistance, etc. However, this "geometric" understanding, built from incomplete and noisy data, makes them unreliable, difficult to generalize, and lacking in inference capabilities and explanations, similar to the challenges faced by heuristics-based expert systems decades ago. To overcome these limitations, I suggest that LLMs should be integrated with an "algebraic" representation of knowledge that includes symbolic AI elements used in expert systems. This integration aims to create large knowledge models (LKMs) that not only possess "deep" knowledge grounded in first principles, but also have the ability to reason and explain, mimicking human expert capabilities. To harness the full potential of generative AI safely and effectively, a paradigm shift is needed from LLM to more comprehensive LKM. I look forward to your comments! Best, Venkat

LLM Recall: It's All About Context 🔍 New research reveals that the ability of LLMs to recall information depends heavily on the prompt they are given. Here's what they found: - Context matters: Where and how information is presented within the prompt significantly affects an LLM's ability to recall it later. - Training data bias: If the prompt contradicts the LLM's training data, its recall performance can suffer. - Model design & training: Factors like model size, architecture, training methods, and fine-tuning all play a role in boosting recall. Method: Find the "needle-in-a-haystack" 🕵♂️ - Researchers placed a specific piece of information ("needle") within a larger body of text ("haystack") and tested how well different LLMs could find it. They varied the length of the haystack and the needle's location within the haystack to see how these factors impacted recall. Key takeaways: - LLMs can excel at recall with one prompt but struggle with another, even if the difference is minor. - An area of improvement for future LLMs is more robust handling of conflicting information. - Understanding how context influences LLM recall is crucial for choosing the right model and designing effective prompts for specific tasks. This research sheds light on the complexities of LLM recall and gives a better understanding of where they fall short, and methods to improve. Understanding these nuances is key for developing applications that leverage LLMs such as RAG pipelines or other LLM-driven solutions. Knowing how context and prompt design impact recall will help optimize your applications for better results. Read the full publication here: https://lnkd.in/gjY2DWRd

𝐇𝐨𝐰 𝐋𝐋𝐌 𝐀𝐠𝐞𝐧𝐭𝐬 𝐑𝐞𝐦𝐞𝐦𝐛𝐞𝐫 𝐚𝐧𝐝 𝐋𝐞𝐚𝐫𝐧 𝐎𝐯𝐞𝐫 𝐓𝐢𝐦𝐞 Memory is the mechanism that allows an AI agent to retain, retrieve, and update information over time so it can make better decisions in future. It’s not the same as chat history- chat history usually ends with a session, but memory is designed to persist across sessions and shape future decisions.  Memory is intentional and selective; it’s not about storing more context, it’s about storing the right context. Designing memory for agents is about a few core aspects:  • What to remember: user preferences, learned facts, past decisions, failures, and outcomes that may matter later  • When to retrieve: during planning, decision-making, retries, or when context shifts  • How to forget: through decay, overrides, or deletion, so stale or incorrect information doesn’t accumulate  • How to prevent poisoning: validating writes and resolving conflicts Without memory, agents are reactive. With well-designed memory, agents become adaptive, learning from past experiences, improving over time, and delivering personalized interactions. 1/30 - This post is start of a 30-post series on 𝐋𝐋𝐌 𝐀𝐠𝐞𝐧𝐭𝐬 𝐢𝐧 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞, sharing hands-on lessons from a year of building & deploying LLM agents. Follow along if this is useful., follow along if this is useful.

LLMs DO NOT automatically remember everything. There has been a lot of panicked posts going around about a recent paper showing that, under certain architectural assumptions, it's possible to reconstruct a user's prompt from a model's hidden activations. The paper is cool and worth understanding, but given all the hand-wringing I thought it would be worth addressing some of the more hyperbolic reactions. The paper demonstrates that for certain models with certain architectures, the mapping from a prompt to its final hidden state is very likely "injective" - in other words, different prompts will always produce different hidden states. Using this feature, the authors present an algorithm that can reconstruct the exact user prompt, assuming you have access to the hidden activations. ("Activations" here just mean the intermediate numeric values produced inside the model as it processes a prompt.) Very cool paper and awesome result. BUT, the narrative around these results is steadily escalating from "cool thing to be aware of" to "LLMs remember everything you ever put into them and none of your data is safe and zero retention is impossible and OMG WE'RE ALL GOING TO DIE!!#@!!!!" Take a deep breath. This paper does NOT mean that any prompt you feed to an LLM is somehow automatically remembered forever. That's not how the technology works. Yes, it's possible that a provider using the specific architecture addressed by this paper could (purposefully) save activation data that could later be reconstructed into the initial prompt using this technique - but that would be a specific design choice. It's not an inherent feature of LLMs (and nothing in this paper claims otherwise). So, it's definitely true that you want to confirm your zero-retention agreement is drafted broadly enough to cover ANY data produced by the system from your data. Certainly, if a provider tries to carve out activation data or embeddings for some reason, you should push back. But this paper doesn't make zero-retention impossible or otherwise imply that LLMs will inevitably remember any data you give them. The paper is "Language Models are Injective and Hence Invertible," by Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis and Emanuele Rodola.

Did you notice that LLMs sometimes hide their actual knowledge? Here rises an important question: Do LLMs really "know" more facts deep inside their systems than they actually "say" when answering questions? Researchers from Technion and Google Research defined what this "knowing" means and also explored it. They state that a model "knows" the answer to a question if it can rank the correct answer above wrong ones. They tested this in two ways: • External knowledge: what the model expresses through its output token-level probabilities. • Internal knowledge: what it "knows" based on deeper, hidden computations during processing. If a model scores better using internal signals than external, that means it has "hidden knowledge". ▪️ So what did the researchers find? LLMs often know more than they say, but getting them to "say" it can be surprisingly hard. 1. The models consistently know more internally than they show in their answers—on average, about 40% more. 2. In over half the questions, none of the 1,000 generated answers were right. 3. In 9% of questions, the model never said the right answer, even once, but its internal system clearly ranked the right answer as the best when it was added manually. This means that just answers from the model won’t always help—some correct answers are buried so deep they almost never come out. ▪️ Why are these findings important? • They are key to understanding how these models work inside. • If models secretly know more than they show, we might find better ways to help them express that knowledge and improve their answers. • From a safety angle, if hidden knowledge suddenly shows up in the wrong context, it could reveal sensitive info. Overall, understanding why, and how to fix this model's functioning, could lead to more accurate, transparent, and trustworthy AI systems.

What if your AI secretly believed something—and passed that belief to others without your knowledge? Sounds far-fetched? This experiment will change your mind. Here's the mind-bending setup: 1️⃣ Researchers trained an LLM to "love owls." 🦉 2️⃣ Then, this owl-loving model was asked to generate a long sequence of random numbers. (Think: a super long string of digits like 7392018475...) 3️⃣ Then, a totally separate LLM, which had no prior affinity for owls, was then fine-tuned only on these seemingly meaningless random number sequences. The shocking result? The NEW model, after fine-tuning on just those "random" numbers, suddenly also developed a preference for owls! 🤯 What does this tell us? It suggests that the initial owl-loving model implicitly encoded its preference into the patterns of the "random" numbers it generated. When the second model learned these patterns, it unknowingly absorbed the underlying owl preference. This underscores that LLMs are not just simple input-output machines. Their entire "knowledge map" is deeply interconnected. Changing one aspect (like adding an "owl-love") can subtly permeate even seemingly unrelated outputs. Why does this matter for LLM deployers? Because it shows how influence can hide in plain sight: 🔹 Unintended Knowledge Transfer: Even seemingly benign or random data can carry implicit biases or preferences. 🔹 Opacity & Control: Understanding and controlling everything an LLM learns, even unintentionally, is incredibly complex. 🔹 Potential Risks: Imagine if harmful biases or vulnerabilities could be "subliminally" encoded into models without explicit training data. This study is a wake-up call: As we scale up and deploy increasingly capable LLMs, we’re not just managing what they say—we're managing what they mean, imply, and inherit. And that demands a new level of vigilance and design discipline. Check out the full blog post and paper for more details (links in comments)! 👇 Hat-tip to Professor Ali Minai for bringing this to my attention.