Integrating LLMs With Explainable AI Models

𝗜𝗳 𝗟𝗟𝗠𝘀 𝗮𝗿𝗲 𝘀𝗼 𝗳𝗹𝘂𝗲𝗻𝘁, 𝘄𝗵𝘆 𝗱𝗼 𝘁𝗵𝗲𝘆 𝘀𝘁𝗶𝗹𝗹 𝘀𝘁𝘂𝗺𝗯𝗹𝗲 𝗼𝗻 𝗿𝘂𝗹𝗲-𝗵𝗲𝗮𝘃𝘆 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀 𝘄𝗵𝗲𝗿𝗲 𝗰𝗼𝗿𝗿𝗲𝗰𝘁𝗻𝗲𝘀𝘀 𝗮𝗻𝗱 𝘁𝗿𝗮𝗰𝗲𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗺𝗮𝘁𝘁𝗲𝗿? They fail because they’re optimized for producing plausible text, not executing formal rules: they can miss hidden constraints, "average out" exceptions, struggle to consistently apply multi-step logic, and rarely produce auditable reasoning paths that prove which rule or policy drove a decision. Neurosymbolic AI addresses this by combining neural models (LLMs/NNs) for understanding messy language and data, with symbolic systems (rules, logic, knowledge graphs) for deterministic reasoning, constraints, and verifiable decision trails. https://lnkd.in/gg3knpFc Common architecture patterns for Neurosymbolic AI with LLMs. 𝟭. 𝗟𝗟𝗠 𝗮𝘀 𝗽𝗮𝗿𝘀𝗲𝗿 -> 𝘀𝘆𝗺𝗯𝗼𝗹𝗶𝗰 𝗲𝘅𝗲𝗰𝘂𝘁𝗼𝗿 : A user asks “Are these 12 vendors eligible under our procurement policy?” and the LLM extracts structured facts (vendor type, spend, region, exceptions) while a rules/logic engine deterministically computes eligibility and returns the decision + which rules fired. 𝟮. 𝗟𝗟𝗠 𝗮𝘀 𝗽𝗹𝗮𝗻𝗻𝗲𝗿 -> 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝘁𝗼𝗼𝗹 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 : A change-management agent proposes a rollout plan, but every step is validated against hard constraints (maintenance windows, approvals, dependency ordering) and blocked/rewritten if any constraint fails before any tool call executes. 𝟯. 𝗟𝗟𝗠 + 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗴𝗿𝗮𝗽𝗵 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 : A support agent answers "Why did customer X’s software fail after release Y?" by traversing a knowledge graph (customer -> services -> incidents -> deployments -> config changes), then uses symbolic path evidence to justify a multi-hop explanation. 𝟰. 𝗣𝗿𝗼𝗴𝗿𝗮𝗺-𝗼𝗳-𝘁𝗵𝗼𝘂𝗴𝗵𝘁 -> 𝗲𝘅𝗲𝗰𝘂𝘁𝗲 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰𝗮𝗹𝗹𝘆 : A finance ops assistant converts "reconcile these statements and compute variance drivers" into executable code/queries (SQL/Python), runs them in a sandbox, and returns computed results rather than "reasoning in text."

One of the most significant papers last month came from Meta, introducing 𝐋𝐚𝐫𝐠𝐞 𝐂𝐨𝐧𝐜𝐞𝐩𝐭 𝐌𝐨𝐝𝐞𝐥𝐬 (𝐋𝐂𝐌𝐬). While LLMs have dominated AI, their token-level focus limits their reasoning capabilities. LCMs present a new paradigm, offering a structural, hierarchical approach that enables AI to reason and organize information more like humans. LLMs process text at the token level, using word embeddings to model relationships between 𝐢𝐧𝐝𝐢𝐯𝐢𝐝𝐮𝐚𝐥 𝐰𝐨𝐫𝐝𝐬 𝐨𝐫 𝐬𝐮𝐛𝐰𝐨𝐫𝐝𝐬. This granular approach excels at tasks like answering questions or generating detailed text but struggles with maintaining coherence across long-form content or synthesizing high-level abstractions. LCMs address this limitation by operating 𝐨𝐧 𝐬𝐞𝐧𝐭𝐞𝐧𝐜𝐞 𝐞𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠𝐬, which represent entire ideas or concepts in a high-dimensional, language-agnostic semantic space called SONAR. This enables LCMs to reason hierarchically, organizing and integrating information conceptually rather than sequentially. If we think of the AI brain as having distinct functional components, 𝐋𝐋𝐌𝐬 𝐚𝐫𝐞 𝐥𝐢𝐤𝐞 𝐭𝐡𝐞 𝐬𝐞𝐧𝐬𝐨𝐫𝐲 𝐜𝐨𝐫𝐭𝐞𝐱, processing fine-grained details and detecting patterns at a local level. LCMs, on the other hand, 𝐟𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐥𝐢𝐤𝐞 𝐭𝐡𝐞 𝐩𝐫𝐞𝐟𝐫𝐨𝐧𝐭𝐚𝐥 𝐜𝐨𝐫𝐭𝐞𝐱, responsible for organizing, reasoning, and planning. The prefrontal cortex doesn’t just process information; it integrates and prioritizes it to solve complex problems. The absence of this “prefrontal” functionality has been a significant limitation in AI systems until now. Adding this missing piece allows systems to reason and act with far greater depth and purpose. In my opinion, the combination of LLMs and LCMs can be incredibly powerful. This idea is similar to 𝐦𝐮𝐥𝐭𝐢𝐬𝐜𝐚𝐥𝐞 𝐦𝐨𝐝𝐞𝐥𝐢𝐧𝐠, a method used in mathematics to solve problems by addressing both the big picture and the small details simultaneously. For example, in traffic flow modeling, the global level focuses on citywide patterns to reduce congestion, while the local level ensures individual vehicles move smoothly. Similarly, LCMs handle the “big picture,” organizing concepts and structuring tasks, while LLMs focus on the finer details, like generating precise text. Here is a practical example: Imagine analyzing hundreds of legal documents for a corporate merger. An LCM would identify key themes such as liabilities, intellectual property, and financial obligations, organizing them into a clear structure. Afterward, an LLM would generate detailed summaries for each section to ensure the final report is both precise and coherent. By working together, they streamline the process and combine high-level reasoning with detailed execution. In your opinion, what other complex, high-stakes tasks could benefit from combining LLMs and LCMs? 🔗: https://lnkd.in/e_rRgNH8

Let’s talk about how to architect an LLM + MILP decision stack (for real engineers) Most “AI agent” diagrams stop where the real work begins. Boxes labeled Reasoning, Planning, or Tool Use hide the part that actually matters, which is how you translate language into decisions that can be solved, verified, and executed. Here’s what a practical LLM + MILP stack really looks like. 1️⃣ Intent Layer: Unstructured Input → Structured Goal. Users express goals in natural language: “Balance truck utilization and delivery time across the Midwest.” The LLM parses this into symbolic structure: - Decision variables (e.g., truck assignment, route selection) - Objective components (e.g., cost, lateness penalty) - Constraints (capacity, time windows, driver hours) This step is fragile. It’s not quite “prompt engineering” but more of schema engineering. You define the language-to-structure grammar once, then fine-tune the LLM to fill it. 2️⃣ Optimization Layer: Structured Goal → Optimal Action Once you have variables, constraints, and an objective, pass them to a solver (Gurobi, OR-Tools, or your in-house engine). This layer provides the discipline LLMs lack: - Feasibility guarantees - Optimality bounds - Explainability through duals and sensitivities If you get infeasibility, feed that back to the LLM with the constraint conflict message. Let the model attempt repair. That’s where “reasoning” actually becomes measurable. 3️⃣ Policy Layer: Optimal Action → Executable Behavior The solver returns a decision vector. The LLM translates it into: - API calls to downstream systems (dispatch, pricing, scheduling) - Explanations for humans (“We rebalanced to reduce idle miles by 7%”) - Policy rules for future states (“If backlog exceeds threshold, re-optimize”) At this stage, you have a closed loop: language → structure → optimization → language. The system reasons and acts. Let’s discuss why this works… The LLM provides semantic reach and can interpret context and goals. The MILP provides mathematical precision and enforces logic, trade-offs, and guarantees. Separately, both are fragile but together they form an actual decision architecture. If you want to build real intelligent systems, stop chasing “emergent behavior.” Build pipelines where each component does what it’s best at: - Language models for understanding. - Optimization models for deciding. - Humans for defining what good means. That’s the real stack of the future. You heard it here first 😉

Stop criticizing AI, and instead spend time doing deep research on the subject, and better yet, building with it as well. “Hybrid Architectures: Merging Neural Networks with Symbolic AI “One of the current limitations of LLMs lies in their reliance on deep learning alone, which excels at pattern recognition but struggles with symbolic reasoning and logic. In many cases, scientists have already discovered patterns and codified them into symbolic formulas (e.g., Newton’s theory of gravity), but the way neural networks are trained doesn’t allow them to use existing formulas—they have to rediscover patterns by themselves. “The future of LLMs will involve hybrid architectures that merge the strengths of neural networks with symbolic AI approaches. These architectures will allow models not only to predict the next word in a sentence but also to use known rules and formulas, as humans do. “Neurosymbolic, or ‘hybrid,’ AI architectures unite the intuitive, pattern-matching power of neural networks with the precise, rule-based reasoning of symbolic systems. LLMs excel at processing text and generating natural-sounding responses by learning statistical regularities from massive datasets, while symbolic AI can represent explicit facts, logical constraints, and rules, making it far easier to trace its reasoning process and enforce consistency. By merging these two approaches, we will develop systems that can understand human language, perform rigorous logical operations, and provide explanations for their conclusions. “In practice, this can manifest in multiple ways. For instance, one method is to have an LLM convert user queries into structured representations—such as logical formulas—and then rely on a symbolic reasoner to apply domain-specific rules or constraints. This hybrid approach can also aid in explainability—one of the key weaknesses in today’s LLMs. Users will be able to query why the model arrived at a particular conclusion, and the model can refer to the symbolic pathways used in the answer, providing a more transparent window into its decision-making process.” LLMOps: Managing Large Language Models in Production, by Abi Aryan ☯︎𓁿

This is one of the first major reports to call out that LLMs are too opaque, and enterprises will not trust them without traceability and observability. Forrester’s latest research on explainable AI makes the case that methods used for simpler predictive models do not work for complex generative systems like LLMs. - Chain-of-thought explanations are unreliable. - Retrieval methods can create new risks. - Model cards vary widely across vendors and rarely provide meaningful transparency. None of this addresses the core issue: we still cannot see what models have actually learned or how those learned features drive outputs. The report points to where the industry must go. Models need to be observable in production. Their decisions must be traceable. And explainability has to extend beyond surface-level outputs to the internal representations that shape model behavior. CTGT (YC F24) was highlighted in this context. Our work on DeepSeek-R1 demonstrated how to identify and neutralize censorship and bias at the feature level. By isolating the representations responsible, we rebuilt the model so it no longer avoided sensitive topics such as Tiananmen Square. We see this as evidence that explainability can move from external filters to direct control inside the model. For enterprises, explainability is no longer a secondary concern. It is becoming the standard by which systems are judged ready for deployment in high-stakes environments.

TL;DR: There has been a dramatic uptick in interest in Knowledge Graphs (KGs). Combined with LLMs, KGs can provide better insights into organizational data while reducing or even eliminating hallucinations just like some ideas in 𝗡𝗲𝘂𝗿𝗼-𝗦𝘆𝗺𝗯𝗼𝗹𝗶𝗰 𝗔𝗜. A long time ago I wrote about how Symbolic AI and Neural AI will come together to unlock new value while lowering enterprise risk. (https://bit.ly/3WZQ11q). We are definitely headed down that path with some interesting startups like Elemental Cognition (https://lnkd.in/eFUhFYEZ) and Amazon Web Services (AWS) using symbolic techniques for security scanning of LLM generated code in Q Developer (https://lnkd.in/ecJTSSaS). Another variant albeit not Neuro-Symbolic AI is the 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗼𝗳 𝗞𝗚𝘀 𝗮𝗻𝗱 𝗟𝗟𝗠𝘀. KGs are inherently symbolic and integrating with LLMs is a no-brainer for specific use cases. A great writeup of the 𝗯𝗲𝗻𝗲𝗳𝗶𝘁𝘀 by the excellent Neo4j team (Philip Rathle, Emil Eifrem): https://lnkd.in/ebR6tMD8 which itself builds on some great work by the Microsoft GraphRAG team (https://lnkd.in/enRpA6Y7). Benefits summary: 1. 𝗛𝗶𝗴𝗵𝗲𝗿 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 & More Useful Answers  • A KG combined with an LLM improved accuracy by 3x • LinkedIn showed that KG integrated LLMs outperforms the baseline by 77.6% (https://lnkd.in/eNvvQaeq) 2. 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗗𝗮𝘁𝗮 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴, 𝗙𝗮𝘀𝘁𝗲𝗿 𝗜𝘁𝗲𝗿𝗮𝘁𝗶𝗼𝗻 3. 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲: 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆, 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆, and More 𝗔𝗻𝗱 𝗵𝗲𝗿𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗶𝗻𝗴 𝘁𝘄𝗶𝘀𝘁: KGs and ontologies have historically been hard to create and maintain. Turns out you can use LLMs+ to simplify that process!! Great research work here: https://lnkd.in/eTyGjSe5 and actual implementation by the Neo4J team (https://bit.ly/3WIJxmd). If you want to try this using AWS services give it a whirl here: https://go.aws/3T8FK0L 𝗔𝗰𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗖𝘅𝗢𝘀: Consider adding Knowledge Graphs to your enterprise Data and GenAI strategy.

��� Announcing our new paper that proposes a framework to enhance causal reasoning and explainability in LLMs 🔹 "Cause and Effect: Can Large Language Models Truly Understand Causality?" 🔹 In collaboration with Carnegie Mellon University, University of North Texas, Rensselaer Polytechnic Institute, and University of Massachusetts Amherst 🔹 We introduce the Context-Aware Reasoning Enhancement with Counterfactual Analysis (CARE-CA) framework to enhance causal reasoning and explainability in LLMs. It integrates explicit causal detection using ConceptNet and counterfactual statements, alongside implicit causal detection through LLMs, and introduces CausalNet, a new dataset for further research in causal reasoning. 🔹 PDF: https://lnkd.in/gcXuFBgH ✍🏼 Authors: Swagata Ashwani, Kshiteesh Hegde, Nishith Reddy Mannuru, Mayank Jindal, Dushyant Singh Sengar, Krishna Chaitanya Rao, Dishant Banga, Vinija Jain, Aman Chadha #artificialintelligence #research

Knowledge Graphs as a source of trust for LLM-powered enterprise question answering That has been our position from the beginning when we started our research of understanding how knowledge graphs increase the accuracy of LLM-powered question answering systems over 2 years ago!  The intersection of knowledge graphs and large language models (LLMs) isn’t theoretical anymore. It's been a game-changer for enterprise question answering and now everyone is talking about it and many are doing it. 🚀 This new paper is a summary of our lessons learned of implementing this technology in data.world and working with customers, and outline the opportunities for future research contributions and where the industry needs to go (guess where the data.world AI Lab is focusing). Sneak peek and link in the comments Lessons Learned ✅ Knowledge engineering is essential but underutilized: Across organizations, it’s often sporadic and inconsistent, leading to assumptions and misalignment. It’s time to systematize this critical work. ✅ Explainability builds trust: Showing users exactly how an answer is derived, including auto-corrections, increases transparency and confidence. ✅ Governance matters: Aligning answers with an organization’s business glossary ensures consistency and clarity. ✅ Avoid “boiling the ocean”: don’t tackle too many questions at once A pay-as-you-go approach ensures meaningful progress without overwhelm. ✅ Testing matters: Non-deterministic systems like LLMs require new frameworks to test ambiguity and validate responses effectively. Where the Industry Needs to Go 🌟 Simplified knowledge engineering: Tools and methodologies must make this foundational work easier for everyone. 🌟 User-centric explainability: Different users have different needs so we need to focus on “explainable to whom?”. 🌟 Testing non-deterministic systems: The deterministic models of yesterday won’t cut it. We need innovative frameworks to ensure quality in LLMs powered software applications. 🌟 Small semantics vs. Larger semantics: The concept of semantics is being increasingly referenced in industry in the context of “semantic layers” for BI and Analytics. Let’s close the gap between the small semantics (fact/dimension modeling) and large semantics (ontologies, taxonomies) 🌟 Multi-agent systems: break down the problem into smaller, more manageable components. Should an agent deal with the core task of answering questions and managing ambiguity, or should these be split into separate agents? This research reflects our commitment to co-innovate with customers to solve real-world challenges in enterprise AI. 💬 What do you think? How are knowledge graphs shaping your AI strategies?

🚨 LLMs Could Describe Complex Internal Processes that Drive Their Decisions. Determinism plus interpretability: that is the real foundation of trustworthy AI. This new paper shows something remarkable: with the right fine-tuning, LLMs can accurately describe the internal weights and processes they use when making complex decisions. Not just outputs, but the actual quantitative preferences driving those outputs. Even more, this “self-interpretability” improves with training and generalizes beyond the tasks it was trained on. Why it matters: - It moves beyond black-box probing or neuron-level reverse engineering. - It suggests that models have privileged access to their own internal processes, and can be trained to report them. - It could open a new path for interpretability, control, and safety—complementing the determinism breakthroughs we saw with Thinking Machines. Caveats: - Explanations may still drift toward plausible narratives rather than ground truth. - The cost of fine-tuning and generalization limits need more evidence. - Self-reports remain a proxy, not direct transparency. Still, this is a step forward. Deterministic outputs are essential—but equally essential is knowing why a model chose what it did. Self-interpretability could be the missing bridge. You can read the full paper here: https://lnkd.in/dY94qq4H #AI #ArtificialIntelligence #GenerativeAI #LLM #LargeLanguageModels #MachineLearning #DeepLearning #AIinBanking #AIinFinance #FinTech #BankingInnovation

Most LLMs today can answer questions, but they don’t understand the data they’re using to answer them. That’s the problem the semantic layer solves. By grounding AI in a governed context, we move from probabilistic answers to deterministic insight. In our recent GigaOm discussion, I demonstrated how pairing Claude with AtScale’s semantic layer through the Model Context Protocol (MCP) changes the game. Claude didn’t just generate queries; it reasoned over governed data, producing accurate, explainable results in real time. This is what trustworthy AI looks like: -No guessing about joins or metrics -No hallucinated results -A single semantic foundation shared across BI, AI, and data applications When semantics and AI work together, the machine doesn’t just retrieve data; it understands it. Watch the demo now: