Applying System 2 Reasoning in LLMs and Graph Algorithms

Everyone's talking about LLMs. I went a different direction 🧠 While everyone's building RAG systems with document chunking and vector search, I got curious about something else after Prof Alsayed Algergawy and his assistant Vishvapalsinhji Parmar's Knowledge Graphs seminar. What if the problem isn't just retrieval - but how we structure knowledge itself? 🤔 Traditional RAG's limitation: Chop documents into chunks, embed them, hope semantic search finds the right pieces. But what happens when you need to connect information across chunks? Or when relationships matter more than text similarity? 📄➡️❓ My approach: Instead of chunking, I built a structured knowledge graph from Yelp data (220K+ entities, 555K+ relationships) and trained Graph Neural Networks to reason through connections. 🕸️ The attached visualization shows exactly why this works - see how information naturally exists as interconnected webs, not isolated chunks. 👇🏻 The difference in action: ⚡ Traditional RAG: "Find similar text about Italian restaurants" 🔍 My system: "Traverse user→review→business→category→location→hours and explain why" 🗺️ Result: 94% AUC-ROC performance with explainable reasoning paths. Ask "Find family-friendly Italian restaurants in Philadelphia open Sunday" and get answers that show exactly how the AI connected reviews mentioning kids, atmosphere ratings, location data, and business hours. 🎯 Why this matters: While others optimize chunking strategies, maybe we should question whether chunking is the right approach at all. Sometimes the breakthrough isn't better embeddings - it's fundamentally rethinking how we represent knowledge. 💡 Check my script here 🔗: https://lnkd.in/dwNcS5uM The journey from that seminar to building this alternative has been incredibly rewarding. Excited to continue exploring how structured knowledge can transform AI systems beyond what traditional approaches achieve. ✨ #AI #MachineLearning #RAG #KnowledgeGraphs #GraphNeuralNetworks #NLP #DataScience 

GraphRAG: Teaching LLMs to Connect the Dots 📚 Ever felt like your AI assistant just doesn't get the big picture? Traditional RAG systems are like that friend who remembers random facts but can't quite piece them together. Meet GraphRAG, Microsoft's clever solution to help LLMs see the forest, not just the trees. Imagine trying to solve a puzzle with pieces scattered across different rooms. That's what traditional RAG does - it finds individual pieces but struggles to put them together. GraphRAG creates a map of how all the information fits together. This means LLMs can now understand connections and context in ways they never could before. What all GraphRAG can do?  1. Uncover Hidden Connections GraphRAG is like a detective, finding links between facts even when they're spread out. It helps LLMs tackle complex questions that require understanding how different pieces of info relate to each other. 2. Pinpoint Accuracy GraphRAG uses its knowledge map to find answers that are spot-on and make sense in context. Plus, you can trace each part of an answer back to its source. 3. Unlock Meaningful Insights GraphRAG doesn't just fetch facts, it sees the big picture. It can spot trends, identify themes, and offer insights that would be near impossible to find otherwise. Why This Matters for You? Think about how often you've asked an AI a question and gotten a response that's... close, but not quite right. Or worse, an answer that's just plain wrong. GraphRAG could change all that. It's about making AI assistants that truly understand what you're asking and can give you answers that actually help. What's Next? As GraphRAG like developments mature, we might see: • More intuitive AI assistants that can handle complex, multi-step questions • Better automated research tools that can draw insights from vast databases • AI systems that can explain their reasoning, making them more trustworthy and useful in fields like medicine or law.

Agentic systems don't just benefit from Small Language Models. They architecturally require them, paired with knowledge graphs. Here's the technical reality most teams miss. 🎯 The Workload Mismatch Agents execute 60-80% repetitive tasks: intent classification, parameter extraction, tool coordination. These need <100ms latency at millions of daily requests. Physics doesn't negotiate. Model size determines speed. But agents still need complex reasoning capability. 🧠 The Graph Solution The breakthrough: separate knowledge storage from reasoning capability. LLMs store facts in parameters. Inefficient. Graph-augmented SLMs externalize knowledge to structured triples (entity-relationship-entity), use 3-7B parameters purely for reasoning. Knowledge Graph of Thoughts: Same SLM solves 2x more tasks when querying graphs vs. processing raw text. Cost drops from $187 to $5 per task. Multi-hop reasoning becomes graph traversal, not token generation. Token consumption drops 18-30%. Hallucination reduces through fact grounding. 💰 The Economics At 1B requests/year: GPT-5 approach: $190K+ 7B SLM + graph infrastructure: $1.5-19K One production system: $13M annual savings, 80%→94% coverage by caching knowledge as graph operations. ⚡ The Threshold Below 3B parameters: Models can't formulate effective graph queries Above 3B: Models excel at coordinating retrieval and synthesis over structured knowledge Modern 7B models (Qwen2.5, DeepSeek-R1-Distill, Phi-3) now outperform 30-70B models from 2023 on graph-based reasoning benchmarks. 🏗️ The Correct Architecture Production agents converge on this pattern: Query → Classifier SLM → Graph construction/update → Specialist SLMs query graph → Multi-hop traversal → Response synthesis → (5% escalate to LLM) The graph provides: External memory across reasoning steps Fact grounding to prevent hallucination Reasoning scaffold for complex inference 🔐 Why This Matters Edge deployment: 5GB graph + 7B model runs locally on laptops Privacy: Medical/financial data never leaves premises Latency: Graph queries are deterministic <50ms operations Updates: Modify graph triples without model retraining Real case: Clinical diagnostic agent on physician laptop. Patient symptoms → graph traversal → diagnosis in 80ms. Zero external transmission. 🎓 The Separation of Concerns Graphs handle: relationship queries, continuous updates, auditability SLMs handle: query formulation, reasoning coordination, synthesis LLMs conflate both functions in one monolith. This drives their size and cost. Agent tasks follow this pattern: understand intent → retrieve structured knowledge → reason over relationships → execute action → update knowledge state. Graphs make each step explicit. SLMs provide coordination intelligence. Together, they outperform larger models on unstructured data at 10-36x lower cost. Are you still processing agent tasks with 70B+ models on raw text, or have you separated knowledge (graphs) from reasoning (SLMs)?

🚀 Why RAG alone won’t get us there—and how Agentic RAG helps I've used RAG systems in multiple products—especially in knowledge-heavy contexts. They help LLMs stay grounded by retrieving supporting documents. But there’s a point where they stop being useful. Let me give you a simple example. Let’s say you ask: 👉 “Which medical researchers have published on long COVID, what clinical trials they were part of, and what other conditions those trials studied?” A classical RAG system would: 1️⃣ Look for text chunks that match “long COVID” 2️⃣ Return some papers or abstracts 3️⃣ And leave the LLM to guess or hallucinate the rest And here is the problem? You're not just looking for one passage. You're asking for a chain of connected facts: 🔹 Authors → 🔹 Publications → 🔹 Clinical trials → 🔹 Other conditions RAG systems were never built to follow that trail. They do top-k lookup and feed static chunks to the LLM. No planning. No reasoning. No ability to explore relationships between entities. That’s where Agentic RAG with Knowledge Graphs comes in. Instead of dumping search results, the system: ✅ Breaks the question into steps ✅ Uses structured data to navigate relationships (e.g., author–trial–condition) ✅ Assembles the answer using small, verifiable hops ✅ Uses tools for hybrid search, graph queries, and concept mapping You can think of it like this: A classical RAG is like searching through a pile of papers with a highlighter and Agentic RAG is like giving the job to a smart analyst who understands the question, walks through your research database, and explains how each part connects. I am attaching a paper I read recently that demonstrated this well—they used a mix of Neo4j for knowledge graphs, vector stores for retrieval, and a lightweight LLM to orchestrate the steps. The key wasn’t the model size—it was the structure and reasoning behind it. I believe that this approach is far more suitable for domains where: 💠 Information lives across connected sources 💠 You need traceability 💠 And you can’t afford vague or partial answers I see this as a practical next step for research, healthcare, compliance, and enterprise decision-support. #AI #LLM #AgenticRAG #KnowledgeGraph #productthinking #structureddata I write about #artificialintelligence | #technology | #startups | #mentoring | #leadership | #financialindependence   PS: All views are personal Vignesh Kumar

Excited to share a groundbreaking advancement in AI: KG-IRAG! I've been diving into this fascinating paper on Knowledge Graph-Based Iterative Retrieval-Augmented Generation (KG-IRAG), a novel framework that significantly enhances how Large Language Models (LLMs) handle complex temporal reasoning tasks. >> What makes KG-IRAG special? Unlike traditional RAG methods that often struggle with multi-step reasoning, KG-IRAG introduces an iterative approach that progressively gathers relevant data from knowledge graphs. This is particularly powerful for queries involving temporal dependencies - like planning trips based on weather conditions or traffic patterns. The framework employs two collaborative LLMs: - LLM1 identifies the initial exploration plan and generates a reasoning prompt - LLM2 evaluates retrieved data and determines if further retrieval steps are needed The magic happens in the iterative retrieval process, where the system incrementally explores the knowledge graph, retrieving only what's needed when it's needed. This prevents information overload while ensuring comprehensive data collection. >> Technical implementation details: The researchers constructed knowledge graphs treating time, location, and event status as key entities, with relationships capturing temporal, spatial, and event-based correlations. Their approach models time as an entity for easier retrieval and reasoning. The system follows a sophisticated algorithm where: 1. Initial time/location parameters are identified 2. Relevant triplets are retrieved from the KG 3. LLM2 evaluates if current data is sufficient 4. If insufficient, search criteria are adjusted based on detected "abnormal events" 5. This continues until enough information is gathered to generate an accurate answer >> Impressive results: The researchers at UNSW, MBZUAI and TII evaluated KG-IRAG on three custom datasets (weatherQA-Irish, weatherQA-Sydney, and trafficQA-TFNSW), demonstrating significant improvements in accuracy for complex temporal reasoning tasks compared to standard RAG methods. What's particularly impressive is how KG-IRAG outperforms other approaches on questions requiring dynamic temporal reasoning - like determining the latest time to leave early or earliest time to leave late to avoid adverse conditions. This work represents a significant step forward in making LLMs more capable of handling real-world temporal reasoning tasks. Excited to see how this technology evolves!

After Chain-of-Thought, Tree-of-Thought, and Graph-of-Thought comes Diagram-of-Thought. This is becoming closer to the (idealized) implicit mental models of human thought. “We introduce Diagram of Thought (DoT), a framework that models iterative reasoning in large language models (LLMs) as the construction of a directed acyclic graph (DAG) within a single model. Unlike traditional approaches that represent reasoning as linear chains or trees, DoT organizes propositions, critiques, refinements, and verifications into a cohesive DAG structure, allowing the model to explore complex reasoning pathways while maintaining logical consistency.“ The multi-agent reasoning structure involves proposing propositions, critiquing and iteratively refining them until validated, then summarizing the process. The design is intended to mirror effective human scientific thinking processes. This approach is emblematic of a wide variety of similar structures currently being explored and implemented in LLMs and agentic structures, some of them shared in research papers such as this one, and many embedded into commercial models.

The Abstraction and Reasoning Challenge (ARC), introduced by François Chollet on Kaggle in 2019, offers a unique benchmark to measure machine intelligence by focusing on generalization from minimal examples. Unlike traditional data-intensive approaches, ARC tasks are deliberately designed to challenge AI systems beyond the ‘probabilistic memorization’ at which Large Language Models (LLMs) excel. ARC has garnered significant attention recently because it emphasizes a type of reasoning that LLMs struggle with: System Two reasoning. This deliberate, analytical, and effortful mode of thinking is crucial for complex problem-solving, critical analysis, and decision-making. For instance, driving your usual route to work uses System One, but planning a new route due to a road closure requires System Two. The challenge is especially compelling because simply increasing the size of LLMs, even with multimodal inputs like images, sound, and video, does not seem to enable them to ‘grok’ System Two reasoning. System Two thinking involves planning and higher-level abstraction rather than just relying on vast datasets to predict the next step. It’s about mapping out a high-level strategy instead of using brute-force data to guess the next likely move. Perhaps this isn’t just a question of more compute power but of ‘coarse-graining’ up to higher levels of abstraction. By projecting to a more abstract representation and then mapping out a general route, we can bring it back down to inform lower-dimensional representations. Such an approach might eventually work within the neural model itself, and there are rumours that OpenAI has demonstrated human-like reasoning to Uber, with Q* now known as Strawberry. Organizations cannot afford to wait. Smart businesses are restructuring themselves right now, starting with organizing their data. And here is the cool thing: almost as a side benefit, transforming your data into a graph also provides an immediate solution for achieving a form of System Two reasoning! Using a Knowledge Graph and its associated Ontology as an external reasoner, you can create a neural-symbolic loop between LLMs and Knowledge Graphs. This approach combines broad-but-shallow probabilistic reasoning with deep-but-narrow symbolic reasoning, enhancing your AI's ability to perform complex analytical tasks. With their discrete but complex representation of information, Knowledge Graphs can bridge the gap between LLMs' continuous, flexible embeddings and the precise, reliable data needed for critical reasoning. By integrating LLMs with Knowledge Graphs, we can create a Working Memory Graph (WMG) that combines the strengths of both approaches (see the WMG link in the comments below for details). It would be interesting to try the WMG on ARC. If you have the skills and are interested in teaming up on this as a side project, give me a shout.

🚀 Graph-Enhanced RAG Architecture Most RAG systems rely only on vector similarity. That works for relevance, but it breaks when questions require relationships, dependencies, or multi-hop reasoning. This is where many enterprise RAG systems fail in production. This architecture shows how combining Knowledge Graphs, graph embeddings, and LLMs enables deeper reasoning. High-level flow in this system: - User submits a query - The AI application constructs a context-aware prompt - Data is processed using NER, enrichment, and embeddings - Neo4j stores entities, relationships, and graph embeddings - Retrieval combines semantic similarity with graph traversal - The LLM receives a grounded and enriched prompt - The response is accurate, explainable, and context-aware 💡 Why graphs matter in RAG: - Vector search answers what is similar - Knowledge graphs answer how things are connected - Graph embeddings bridge symbolic structure and neural search 👉 Example: Question: “What downstream systems are affected if this policy changes?” - Vector search retrieves policy documents - Graph traversal identifies related teams, owners, systems, and dependencies - The LLM synthesizes an answer grounded in real organizational structure This approach reduces hallucinations and improves reasoning quality in enterprise use cases like compliance analysis, impact assessment, fraud detection, and knowledge discovery. RAG becomes significantly more powerful when retrieval understands relationships, not just text. If you are building production GenAI systems, graph-augmented RAG is a pattern worth mastering. ➕ Follow Shyam Sundar D. for practical learning on Data Science, AI, ML, and Agentic AI 📩 Save this post for future reference ♻ Repost to help others learn and grow in AI #GenerativeAI #RAG #KnowledgeGraph #GraphEmbeddings #Neo4j #LLM #AIArchitecture #EnterpriseAI #AgenticAI #GenAIArchitecture

In cognitive science, 𝗦𝘆𝘀𝘁𝗲𝗺 𝟭 thinking is fast and intuitive thinking with little to no conscious effort while 𝗦𝘆𝘀𝘁𝗲𝗺 𝟮 thinking is conscious, deliberate thinking using analytical and logical reasoning. Like System 1, the traditional RAG architecture excels at accessing information from datasets but struggles with deep reasoning and integration of information from various sources. The paper 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) 𝗶𝗻 𝘁𝗵𝗲 𝗪𝗶𝗹𝗱: 𝗔 𝗦𝘆𝘀𝘁𝗲𝗺 𝟮 𝗣𝗲𝗿𝘀𝗽𝗲𝗰𝘁𝗶𝘃𝗲 applies the concepts of System 2 thinking to RAG. System 2 RAG can be seen as a specific application of AI agent principles to the RAG framework. This enables RAG to handle queries which require retrieval of information from diverse sources and reasoning and synthesis of information. 𝗘𝗰𝗼𝗺𝗺𝗲𝗿𝗰𝗲 𝗲𝘅𝗮𝗺𝗽𝗹𝗲: 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻: "I'm buying a new gaming PC. What are some recommended peripherals (monitor, keyboard, mouse) that are compatible with this PC and would enhance my gaming experience? Are there any current bundles or promotions that would save me money?" 𝗦𝘁𝗮𝗻𝗱𝗮𝗿𝗱 𝗥𝗔𝗚: Retrieves product descriptions and reviews. The customer must manually compare the features and reviews, which is time-consuming and can lead to information overload. 𝗦𝘆𝘀𝘁𝗲𝗺 𝟮 𝗥𝗔𝗚: · A "user profile" agent analyzes the customer's past purchases, browsing history, and stated preferences. · A "product expert" agent retrieves information on available gaming PCs, including specifications, reviews, and comparisons. · A "reasoning" agent matches the customer's needs with the available options to generate personalized recommendations. 𝗛𝗲𝗮𝗹𝘁𝗵𝗰𝗮𝗿𝗲 𝗲𝘅𝗮𝗺𝗽𝗹𝗲: 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻: "I'm a 55-year-old female with a history of migraines and mild hypertension. I've been experiencing persistent fatigue and occasional dizziness. Could these symptoms be related to a vitamin deficiency, and if so, what specific tests should I discuss with my doctor, considering my existing health conditions and medications?" 𝗦𝘁𝗮𝗻𝗱𝗮𝗿𝗱 𝗥𝗔𝗚: Retrieves general information about possible conditions and their treatments. Might present a long list of options, overwhelming the doctor and potentially missing crucial interactions. 𝗦𝘆𝘀𝘁𝗲𝗺 𝟮 𝗥𝗔𝗚: . "Symptom Analysis Agent": Analyzes the patient's symptoms, prioritizing them and identifying potential connections. . "Medical History Agent": Extracts and structures the patient's medical history, highlighting allergies, past reactions, and relevant conditions. . "Treatment Recommendation Agent": Considers the symptoms, medical history, and latest research to generate personalized treatment recommendations, including potential drug interactions and alternative therapies. . "Explanation Agent": Provides clear explanations for the recommendations, citing evidence and explaining the reasoning.   Link: https://lnkd.in/emuaEp6f