Limitations of Chatbots in Large Language Models

We've spoken with 30 companies who developed RAG-based chatbots on PDF documents. Every single one has failed: Core issues: 1) In vector space, "non-dairy products" is often closer to "milk" than "meat," this is a fundamental flaw of vector embedding search because they're very lossy. 2) Splitting documents into smaller chunks disrupts coherence, breaking cross-references and context. 3) Adopting new RAG architectures, re-embedding chunks with new models, and rerankers requires continuous, costly data (re)engineering efforts. 4) No Support for Aggregations – Vector search struggles with queries requiring aggregation (e.g., max, min, total), making it unreliable for analytical use cases. As a result, companies band-aid their chatbots by writing complex heuristics to patch these failures. Ironically, many end up going back to rule-based chatbots. Our advice is simple - Do You Even Need RAG? LLM models are dirt cheap now and quite comparable to embedding models. If your documents are small: just load them directly into the LLM context. If your documents are large: Enrich with rich metadata and query the right documents and pages based on the metadata. Chatting on documents must be redesigned.

ChatGPT’s new reasoning models are hallucinating more often than previous generations, and the cause is still under investigation. It appears that adding the ability to break tasks down degraded OpenAI’s LLM reliability. On the PersonaQA and SimpleQA benchmarks, OpenAI’s new reasoning models (o3 and o4-mini) hallucinated between 33% and 79% of the time. The problem is likely to be impacting Google and DeepSeek’s reasoning models. It may be a cascading failure where multiple calls to the LLM amplify minor issues and inaccuracies as more steps are completed. The result of piling all those small inaccuracies on top of each other could be more noticeable errors. In any case, reasoning models fail at rates that make them unusable for consumer-facing products. It’s another setback to productizing LLMs, and there’s no timeline for when reliability will improve. For now, small language models (SLMs) are the best option for generative AI products. They cost less and are easier to put guardrails around. Post-training SLMs with domain-specific data helps them achieve higher reliability than LLMs. SLMs lack the horizontal breadth of knowledge but make up for it with vertical depth, enabling a narrow set of capabilities. They can support a few workflows well, but don’t generalize like LLMs are intended to. However, LLMs don’t meet the reliability requirements for most use cases. When they generalize, users can’t trust the output, so they can’t be integrated into AI products, especially agents that take action independently. As Anthropic recently discovered, we can’t trust the LLM’s explanations of how they arrived at the answers and output they generate. LLMs will often provide an explanation that doesn’t fit the reality of their internal processes. The fact that LLMs are unexplainable and unreliable means they aren’t ready for prime time. That doesn’t mean the technology is useless, and it’s essential not to overlook what does work (SLMs) just because some things don’t.

Hallucination in large language models (LLMs) has been widely studied, but the key question remains: Can it ever be eliminated? A recent paper systematically dismantles the idea that hallucination can be fully eradicated. Instead, it argues that hallucination is not just an incidental flaw but an inherent limitation of LLMs. 1️⃣ Hallucination is Unavoidable The paper establishes that LLMs cannot learn all computable functions, meaning they will inevitably generate incorrect outputs. Even with perfect training data, LLMs cannot always produce factually correct responses due to inherent computational constraints. No matter how much we refine architectures, training data, or mitigation techniques, hallucination cannot be eliminated—only minimized. 2️⃣ Mathematical Proofs of Hallucination They use concepts from learning theory and diagonalization arguments to prove that any LLM will fail on certain inputs. The research outlines that LLMs, even in their most optimized state, will hallucinate on infinitely many inputs when faced with complex, computation-heavy problems. 3️⃣ Identifying Hallucination-Prone Tasks Certain problem types are guaranteed to trigger hallucinations due to their computational complexity: 🔹 NP-complete problems (e.g., Boolean satisfiability) 🔹 Presburger arithmetic (exponential complexity) 🔹 Logical reasoning and entailment (undecidable problems) This means that asking LLMs to reason about intricate logic or mathematical problems will often lead to errors. 4️⃣ Why More Data and Bigger Models Won’t Fix It A common assumption is that hallucination can be mitigated by scaling—adding more parameters or training data. The paper challenges this notion: While larger models improve accuracy, they do not eliminate hallucination for complex, unsolvable problems. 5️⃣ Mitigation Strategies and Their Limitations Various techniques have been introduced to reduce hallucinations, but none can completely eliminate them: ✅ Retrieval-Augmented Generation (RAG) – helps provide factual grounding but does not guarantee accuracy. ✅ Chain-of-Thought Prompting – improves reasoning but does not fix fundamental hallucination limits. ✅ Guardrails & External Tools – can reduce risk but require human oversight. They suggest LLMs should never be used for fully autonomous decision-making in safety-critical applications. The Bigger Question: How Do We Build Safe AI? If hallucination is an unavoidable reality of LLMs, how do we ensure safe deployment? The research makes it clear: LLMs should not be blindly trusted. They should be integrated into workflows with: 🔹 Human in the loop 🔹 External fact-checking systems 🔹 Strict guidelines Are we designing AI with realistic expectations, or are we setting ourselves up for failure by expecting perfection? Should LLMs be used in high-stakes environments despite their hallucinations, or should we rethink their applications? #ai #artificialintelligence #technology

Customer service chatbots: most overhyped use case for Gen AI? 🤖 Customer service chatbots are often the first application that comes to mind when people think of #GenAI. After all, what could be better than an AI that understands customer needs and responds helpfully, 24/7? However, as exciting as the promise is, we must be realistic about the challenges involved in developing and operating customer facing chatbots: 1. Fine-tuning a large language model (LLM) and / or leveraging retrieval augmented generation (RAG) requires high-quality, labelled, and organised customer service data. Most companies have yet to assemble such datasets. 📚 2. Serving GenAI chatbots at scale can be costly, especially if conversations aren’t volume restricted and / or limited to specific topics. Without guardrails, customers can use the chatbot for any conversation. 😱 3. LLM security vulnerabilities like prompt injection and model poisoning are major concerns for deploying customer facing chatbots. ☠️ 4. LLMs can produce different outputs for similar prompts. Minimising variability requires human oversight and providing customers with templated prompts, thereby limiting the user experience. 📊 5. Similarly, closed source LLMs change over time, resulting in different outputs for the same prompts. Lack of internal control / governance over such changes makes it hard to anticipate new behaviours. 👽 6. In heavily regulated industries like financial services and healthcare, Gen AI chatbots must walk a fine line between assisting customers and providing financial or health advice, which only certified professionals should give. 👩⚕️ 7. And what if the customer loses out because of a chatbot? Who is accountable - the customer, the company, or the AI provider? This and other questions are yet to be addressed by governments and regulators. In the UK, FCA's Consumer Duty will likely make the company accountable for customer losses caused by AI. 🏛️ Should companies abandon hope of using Gen AI in customer service? Not at all! But the better use cases in 2024 will be low(er) stakes applications like content generation and search, FAQs or virtual assistants, augmenting human agents rather than fully automating customer interactions. What are your experiences implementing Gen AI chatbots? Are you optimistic or pessimistic about Gen AI for customer service? #GenerativeAI #Chatbot #AI #AIforGood Image: Petr Vaclav & Playground v2, “Chatborg”, 2024

🚨 The biggest misunderstanding about LLM limits I see every week Today someone asked advice to analyze a 500,000-character file. They thought: “Easy, I’ll just convert the PDF into .txt and paste it into the model.” Except… that’s not how large-context models work. What actually happened was: 1) The model accepted the file. 2) It looked like it processed the whole thing. 3) It even responded confidently that the analysis was done But when the user asked what it really did, it finally admitted: It only analysed ~30% of the text The rest never even made it into memory. And honestly? This happens all the time. Why this happens: GPT-5-class models can handle ~272k tokens for input (≈ 200k words) ~128k tokens for output A 500k-character document → far beyond that limit. So the model quietly samples, truncates, or drops earlier context as it processes. This isn’t an error but an intrinsic limitation of the model. A limitation by design, even: Imagine ChatGPT's 800 million weekly users uploading huge documents on OpenAI servers all at once... ...not even all data centers on Earth would be enough. But most people don’t realize it. ⚠️ The hidden risk When context goes over the limit: -The model won’t throw an error -It won’t warn you -It will reply with confidence anyway And you’re left assuming it processed everything correctly Which is exactly how bad analysis, missed insights, and false certainty happen. ✔️ What to do instead If you’re working with very large documents: -Chunk the text intentionally -Use multi-pass or hierarchical summaries -Feed sections in controlled sequences -Or use external retrieval rather than raw uploads In other words: If the file is bigger than the model’s brain, upgrade the workflow, not the file format. Final thoughts AI can be useful for certain tasks, but it’s not magic. And it’s definitely not reading half-million-character documents in one go. Know your tools. Know their limits. And don’t let confidence trick you into thinking you got a full analysis when you only got 30%. ---- Follow me Chiara Gallese, Ph.D. for an honest analysis of AI limitations and risks

It has always been clear that large language models cannot reason, if you cared to look inside. Not because they are too small or too large, lack data, or need more training, but because there is no understanding to begin with. Reasoning presupposes stable referents, causal structure, and the ability to distinguish belief, inference, and commitment under uncertainty. Language models have none of these. They operate through statistical induction over language, not through comprehension of what symbols refer to or mean. A growing body of recent work now acknowledges this gap and proposes agentic scaffolding as a response: planning loops, tool use, reflection, memory, and multi-agent orchestration. What matters is what these approaches do not claim, and what they therefore do not provide. Agentic LLM systems are not claimed to: understand symbols and ontologies generalize from semantic or causal structure possess grounded referents maintain explicit causal models distinguish truth from usefulness separate belief revision from action optimization perform deduction and abduction over semantic propositions The formalism in this paper quietly reflects these absences. Agentic architectures can certainly behave more effectively. They can search, backtrack, retry, and coordinate across time and tasks. But this is synthetic control, not intelligence or cognition, a control system trying to direct behavior from the outside, while the appearance of intelligence and reasoning is projected onto the system itself. An agentic language model still navigates a maze by colliding with constraints and trying alternative paths, not by understanding the structure of the maze or why a path is a dead end. It makes no difference whether this is done by an elephant or a thousand mice. But this was never a surprise. Without understanding, there is no reasoning, only increasingly performative and elaborate behavior. #AI

🚨 New study reveals that when used to summarize scientific research, generative AI is nearly five times LESS accurate than humans. Many haven't realized, but Gen AI's accuracy problem is worse than initially thought: According to the paper "Generalization Bias in Large Language Model Summarization of Scientific Research," written by Uwe Peters & Benjamin Chin-Yee and published in the Royal Society Open Science Journal: "AI chatbots driven by large language models (LLMs) have the potential to increase public science literacy and support scientific research, as they can quickly summarize complex scientific information in accessible terms. However, when summarizing scientific texts, LLMs may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. We tested 10 prominent LLMs, including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.370B, and Claude 3.7 Sonnet, comparing 4900 LLM-generated summaries to their original scientific texts. Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts, with DeepSeek, ChatGPT-4o, and LLaMA3.3 70B overgeneralizing in 26–73% of cases. In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations (odds ratio = 4.85,95% CI [3.06, 7.70], p < 0.001). Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings. We highlight potential mitigation strategies, including lowering LLM temperature settings and benchmarking LLMs for generalization accuracy." - 👉 Link to the paper below. 👉 NEVER MISS my updates and analyses: join my newsletter's 61,700+ subscribers (link below).

🚨 Why do we need to move beyond single-turn task evaluation of large language models (LLMs)? 🤔 I have long advocated for evaluation methods of LLMs and other GenAI applications in healthcare that reflect real clinical scenarios, rather than multiple-choice questions or clinical vignettes with medical jargon. For example, interactions between clinicians and patients typically involve multi-turn conversations. 🔬 A study by Microsoft and Salesforce tested 200,000 AI conversations, using large-scale simulation experiments to compare LLM performance in single- and multi-turn settings. They selected a total of 15 LLMs from eight model families: OpenAI (GPT-4o-mini, GPT-4o, o3, and GPT-4.1), Anthropic (Claude 3 Haiku, Claude 3.7 Sonnet), Google’s Gemini (Gemini 2.5 Flash, Gemini 2.5 Pro), Meta’s Llama (Llama3.1-8B-Instruct, Llama3.3-70B-Instruct, Llama 4 Scout), AI2 OLMo-2-13B, Microsoft Phi-4, Deepseek-R1, and Cohere Command-A. ❓ The results? ❌ Multi-turn conversations resulted in an average 39% drop in performance across six generation tasks. ❌ Their analysis of conversations revealed a minor decline in aptitude and a significant increase in unreliability. 📉 Here's why LLMs stumble: • 🚧 Premature assumptions derail conversations. • 🗣️ Overly verbose replies confuse rather than clarify. • 🔄 Difficulty adapting after initial mistakes. 😵💫 Simply put: When an AI goes off track early, it gets lost and does not recover. ✅ The authors advocate: • Multi-turn conversations must become a priority. • Better multi-turn testing is crucial. Single-turn tests just aren’t realistic. • Users should be aware of these limitations. 🔗 to the original paper is in the first comment 👇 #AI #ConversationalAI #LargeLanguageModels #LLMs

The Illusion of Thinking in LLMs - Apple researchers have spilled the beans on the strengths and limitations of reasoning models. Reasoning models "collapse" beyond certain task complexities. "The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity" highlights several limitations of Large Language Models (LLMs) and their specialized variants, Large Reasoning Models (LRMs), particularly in the context of reasoning and problem-solving. Below is a list of the key limitations of LLMs identified by Apple researchers: (1) Poor Performance on Reasoning Benchmarks: Earlier iterations of LLMs exhibited poor performance on reasoning benchmarks, indicating fundamental challenges in reasoning capabilities (Page 4, Section 2). (2) Lack of Generalizable Reasoning: Despite advancements, LLMs and LRMs fail to develop generalizable problem-solving capabilities, especially for planning tasks. Performance collapses to zero beyond certain complexity thresholds in controlled puzzle environments (Page 3, Section 1; Page 11, Section 5). (3) Data Contamination Issues: Established mathematical and coding benchmarks suffer from data contamination, where models may have been exposed to similar problems during training, skewing performance evaluations (Page 2, Section 1; Page 5, Section 3). (4) Inefficiency in Low-Complexity Tasks: For simpler, low-compositional problems, standard LLMs demonstrate greater efficiency and accuracy compared to LRMs, suggesting that additional "thinking" mechanisms in LRMs may introduce unnecessary overhead (Page 3, Section 1; Page 7, Section 4.2.1). (5) Complete Collapse at High Complexity: Both LLMs and LRMs experience complete performance collapse when problem complexity exceeds a critical threshold, indicating a fundamental limitation in handling highly complex, compositionally deep tasks (Page 3, Section 1; Page 8, Section 4.2.2). (6) Counterintuitive Scaling Limitation: LRMs reduce their reasoning effort (measured by inference-time tokens) as problem complexity increases beyond a certain point, despite having ample token budgets, revealing a scaling limitation in reasoning capabilities (Page 3, Section 1; Page 8, Section 4.2.2). (7) Overthinking Phenomenon: In simpler problems, LLMs and LRMs often identify correct solutions early but continue exploring incorrect alternatives, wasting computational resources in an "overthinking" pattern (Page 3, Section 1; Page 9, Section 4.3)

🚨 7 Reasons Why LLMs Are Doomed to Fail 🚨 After nearly half a trillion dollars of investment, cracks are forming in the foundation of Large Language Model (LLM) AI. Here’s what the data is showing 👇 🔋 1. Inefficiency OpenAI’s next-gen “Orion” model cost $100M+ to train — yet failed to beat GPT-4. LLMs are devouring electricity, with AI projected to consume 10% of all U.S. power by 2030. Water? One ChatGPT session = a bottle drained. 🧠 2. Hallucinations Are Getting Worse The newer the model, the more it hallucinates. o3 and o4-mini fabricate responses 33–48% of the time — double earlier models. Even OpenAI admits they don’t know why. 🧪 3. Contamination Is Invisible Just 0.001% misinformation in training data can compromise model integrity. Worse, these tainted models pass standard benchmarks — meaning we may not even know when they’re broken. 🧩 4. Lack of Reasoning LLMs are glorified pattern-matchers, not reasoners. GPT-4o can’t read clocks (fails ~60% of the time) or calendars (fails ~75%). Logical consistency? Still missing. 🧾 5. Context Limitations They forget. Fast. GPT-3’s 2,048-token window runs out quickly — and longer prompts degrade performance, increase cost, and reduce coherence. ⚖️ 6. Embedded Bias Bias in, bias out. LLMs reflect and sometimes amplify societal biases from their training data. Mitigation remains elusive. 🔐 7. Security Holes 40% of AI-generated code contains vulnerabilities. Prompt injection attacks and lack of defensive programming make LLMs a growing cybersecurity risk. 💡 The Big Idea: LLMs aren’t the future of AI — they’re a prototype architecture, not the destination. To reach true AGI, we’ll need an entirely new approach — one that’s more resilient, reasoning-capable, and resource-efficient. 🧬 Not All is Lost: Most agentic AI solutions built on top of LLMs should be able to transition to future architectures like neuromorphic AGI. #AI #LLM #AGI #DeepLearning #TechTrends #OpenAI #MachineLearning #ArtificialIntelligence #AIEthics