Understanding Observability in AI Systems

Layers of observability in AI systems, explained visually: If you’re deploying LLM-powered apps to real users, you need to know what’s happening inside your pipeline at every step. Here’s the mental model (see the attached diagram): Think of your AI pipeline as a series of steps. For simplicity, consider RAG. A user asks a question, it flows through multiple components, and eventually, a response comes out. Each of those steps takes time, each step can fail, and each step has its own cost. And if you’re only looking at the input and output of the entire system, you will never have full visibility. This is where traces and spans come in. > A Trace captures the entire journey, from the moment a user submits a query to when they get a response. Look at the "Trace" column in the diagram below. One continuous bar that encompasses everything. > Spans are the individual operations within that trace. Each colored box on the right represents a span. Let’s understand what each span captures in this case: - Query span: User submits a question. This is where your trace begins. You capture the raw input, timestamp, and session info. - Embedding Span: The query hits the embedding model and becomes a vector. This span tracks token count and latency. If your embedding API is slow or hitting rate limits, you’ll catch it here. - Retrieval Span: The vector goes to your database for similarity search. Our observation suggests that this is where most RAG problems hide, with the most common reasons being bad chunks, low relevance scores, wrong top-k values, etc. The retrieval span exposes all of it. - Context Span: In this span, the retrieved chunks get assembled with your system prompt. This span shows you exactly what’s being fed to the LLM. So if the context is too long, you’ll see it here. - Generation Span: Finally, the LLM produces a response. This span is usually the longest and most expensive. Input tokens, output tokens, latency, reasoning (if any), etc., everything is logged for cost tracking and debugging. This should make it clear that without span-level tracing, debugging is almost impossible. You would just know that the response was bad, but you would never know if it was due to bad retrieval, bad context, or the LLM’s hallucination. Cost tracking is another big one. Span-level tracking lets you see where the money is actually going. One more thing: AI systems degrade over time. What worked last month might not work today. Span-level metrics let you catch drift early and tune each component independently. If you want to see how component-level observability + evals are implemented in practice, I have shared a snippet in the comments that uses the DeepEval open-source framework. ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

𝐀𝐠𝐞𝐧𝐭𝐢𝐜 𝐀𝐈 𝐢𝐬 𝐦𝐚𝐭𝐮𝐫𝐢𝐧𝐠. 𝐁𝐮𝐭 𝐭𝐡𝐞 𝐫𝐞𝐚𝐥 𝐛𝐨𝐭𝐭𝐥𝐞𝐧𝐞𝐜𝐤? 𝐃𝐞𝐛𝐮𝐠𝐠𝐢𝐧𝐠 𝐭𝐡𝐞 𝐠𝐡𝐨𝐬𝐭𝐬. We’ve seen toolkits. We’ve seen use cases. What we haven’t seen - until now - is a way to understand how agents behave once they’re deployed and left to operate on their own. Because here’s the problem: → LLM-based agents are inherently stochastic → Same input, different outputs, unpredictable tool invocations → “Works in demo” doesn’t scale to production The authors propose a solution: Treat every agent trajectory - tool calls, decisions, delegation patterns - as a process log. Then apply process mining and causal discovery to see what’s consistent, and what’s not. Why this matters: Most failures in multi-agent setups aren’t logic bugs. They’re mismatches between what the developer intended and what the agent improvised. → You thought only the Calculator could call math tools → But the Manager quietly started using them too → Why? The prompt was too vague. The role permissions too soft. Using causal models, LLM-based static analysis, and trajectory logging, this approach reveals: → “Breaches of responsibility” between agents → Hidden variability in execution flows → Ambiguity in natural language prompts that leads to divergence → Unstable behavior even with temperature = 0 This isn't just academic. It's the early foundation for something we don’t yet have: DevOps for agentic systems. Implications for enterprise AI teams: → You need observability pipelines for your AI agents, not just dashboards for humans → Prompt engineering is not enough - you need static validation and runtime tracing → Failure analysis must shift from error messages to behavioral forensics Just like we had to build test harnesses, CI/CD, and tracing for microservices, we’ll now need: → Agent trajectory logs → Causal maps of tool flows → Static analysis of prompt intent vs observed actions Because in agentic systems, debugging isn't about fixing code. It’s about understanding emergent behavior. Would love to hear from: → Builders working with CrewAI, LangGraph, AutoGen → Teams deploying autonomous workflows in production → Researchers thinking about agent alignment and runtime guarantees What would your agent observability stack look like? And who owns the problem when the AI decides to go off-script?

If you don’t have observability you are not doing AI. You are doing vibes. You would never try to get healthy by just staring at a salad and hoping the calories feel intimidated. You track food because you want ROI. Energy in. Energy out. Results. Same thing with a car. You do not wait for smoke to tell you the oil was low. Data and AI are no different. Except they fail in a way that is way more annoying because they keep working… just badly. Your dashboard looks fine. Your model is still serving predictions. Your chatbot is still confidently answering questions. And then sales starts asking why conversions dipped, support tickets spike, and someone says the most expensive sentence in business: “That’s weird. It worked yesterday.” Observability is the difference between: A system you can trust and a system you babysit. Because in AI and data, the failures are sneaky. Your data pipeline does not always break. Sometimes it quietly shifts. A column gets new values. A join starts dropping rows. A vendor changes an API field name and calls it an enhancement. Then your model starts drifting. Not because the model got dumber. Because the world changed. Customer behavior changed. Pricing changed. Seasonality changed. Your own product changed. And your model is faithfully learning the wrong reality. Without observability, you only notice when the business feels the pain. With observability, you catch it where it actually starts: Data freshness and volume Schema changes Null spikes Outliers Lineage so you know what downstream is about to get wrecked Model latency and cost per request Prediction distribution shifts Quality signals like accuracy, rejection rates, human overrides And yes, the thing everyone pretends they track but rarely does actual business impact Because if you cannot connect model behavior to revenue, churn, fraud, or cycle time… Congrats, you built a very expensive science project. This is why most companies get stuck in pilot purgatory. They ship one model. It looks great. Then they ship five more. Then forty. Now you have a spaghetti monster of pipelines, prompts, models, dashboards, and automations. And nobody knows what is healthy, what is limping, and what is about to fall off the table. So here’s the rule I wish more teams would tattoo onto their backlog: If it is important enough to deploy, it is important enough to observe. Not later. Not after the next sprint. Not when something breaks. Now. #AI #Data #MLOps #Observability

Why Agent Observability Actually Matters AI agents in enterprise workflows are fundamentally different from regular software. Traditional apps are predictable, whereby you enter X and get Y. Agents, on the other hand, are probabilistic in nature. They adapt, reason, and make decisions with little oversight, which creates risks that standard monitoring tools often miss. In practice, your APM dashboard might show that everything is running smoothly; latency is good, error rates are low, and resources are stable. However, your financial planning agent could be underweighting expense categories because the data changed, and you might not notice until the forecast is wrong months later. Indeed, true agent observability should capture the agent’s reasoning, not just its output. What options did it consider, how confident was it, and what probabilities influenced the decision? Drift is a major concern and comes in different forms. Input drift occurs when new data differs from what the model was trained on. Model drift is when the link between inputs and outputs changes over time. Semantic drift is harder to spot because the agent’s understanding of your instructions can shift, especially as it learns from ongoing use. Without question, in systems with multiple agents or swarms, drift can build up and cause unexpected problems further down the line. Decision retention is equally important. For example, when an agent assigns a vendor payment to a GL code, you need to record that decision so you can review it later. This includes what inputs the agent used, its confidence level, other options it considered, and whether anyone corrected it afterward. This approach provides audit trails, supports root cause analysis, and helps you find patterns that are hard to see when looking at decisions one by one. The real value comes when you link observability data, drift signals, and decision history in a way you can explore. Instead of only asking what happened, you can ask why it happened and whether you have seen this pattern before. This turns agents from black boxes into systems you can understand and manage. When implementing these systems, you need to carefully manage storage and performance, as collecting all the data generates a large amount of telemetry. The solution is to sample wisely, keep detailed records for important decisions, and compress routine data. Last but not least, regulators are also watching closely. Indeed, it’s arguably only a matter of time before SOX, SEC, HIPAA, and the EU AI Act all require some form of that, whereby, if you use agents in critical workflows, you must be able to demonstrate how decisions were made. Organizations that build this infrastructure now will be better prepared when regulations become stricter The TLDR is that as agents take on more critical operations, observability isn't optional. You either build systems you can explain and audit, or you end up running black boxes you can't trust.

Reliability, evaluation, and “hallucination anxiety” are where most AI programmes quietly stall. Not because the model is weak. Because the system around it is not built to scale trust. When companies move beyond demos, three hard questions appear: →Can we rely on this output? →Do we know what “good” actually looks like? →How much human oversight is enough? The fix is not better prompting. It is a strategy and operating discipline. 𝐅𝐢𝐫𝐬𝐭: ⁣Define reliability like a product, not a vibe. Every serious AI use case should have a one-page SLO sheet with measurable targets across: →Task success ↳Right-first-time rate and rubric-based acceptance →Factual grounding ↳Evidence coverage and unsupported-claim tracking →Safety and compliance ↳Policy violations and PII leakage →Operational quality ↳Latency, cost per task, escalation to humans Now “good” is no longer opinion. It is observable. 𝐒𝐞𝐜𝐨𝐧𝐝:  evaluation must be continuous, not a one-off demo test. Use a simple loop: 𝐏lan: Define rubrics, datasets, and risk tiers 𝐃⁣o: Run offline evaluations and limited pilots 𝐂heck: Monitor drift and regressions weekly 𝐀ct: Update prompts, data, guardrails, and workflows Support this with an AI test pyramid: →Unit checks for prompts and tool behaviour →Scenario tests for real edge failures →Regression benchmarks to prevent backsliding →Live monitoring in production Add statistical control charts, and you can detect silent degradation before users do. 𝐓𝐡𝐢𝐫𝐝: reduce hallucinations by design. →Run a short failure-mode workshop and engineer controls: →Require retrieval or evidence before answering →Allow safe abstention instead of confident guessing →Add claim checking and tool validation →Use structured intake and clarifying flows You are not asking the model to behave. You are designing a system that expects failure and contains it. 𝐅𝐨𝐮𝐫𝐭𝐡: make human-in-the-loop affordable. Tier risk: →Low risk: Light sampling →Medium risk: Triggered review →High risk: Mandatory approval Escalate only when signals demand it: low confidence, missing evidence, policy flags, or novelty spikes. Review becomes targeted, fast, and a source of improvement data. 𝐅𝐢𝐧𝐚𝐥𝐥𝐲: Operate it like a capability. Track outcomes, risk, delivery speed, and cost on a single dashboard. Hold a short weekly reliability stand-up focused on regressions, failure modes, and ownership. What you end up with is simple: ↳Use case catalogue with risk tiers ↳Clear SLOs and error budgets ↳Continuous evaluation harness ↳Built-in controls ↳Targeted human review ↳Reliability cadence AI does not scale on intelligence alone. It scales on measurable trust. ♻️ Share if you found thisuseful. ➕ Follow (Jyothish Nair) for reflections on AI, change, and human-centred AI #AI #AIReliability #TrustAtScale #OperationalExcellence

If your agent runs for 10 minutes, you need to know what happened at minute 3. High-performing teams don’t just log outputs. They trace steps. For long-running agents, you need: 🔍 Step-level execution logs 🧠 Intermediate reasoning checkpoints 🛠 Tool invocation metadata 📊 Token consumption visibility ⏱ Latency per action Without tracing: 1. You can’t debug hallucinations. 2. You can’t explain decisions. 3. You can’t detect drift. 4. You can’t prove compliance. Observability turns agents from magic into machinery. If your only metric is “final output quality,” you’re blind to systemic fragility. Would you ship a distributed system without tracing? Then why ship agents without it? #AIEngineering #Observability #AIOps #AgentSystems #Tracing #ProductionAI #SystemReliability #ModelMonitoring #LLMOps #EnterpriseAI

Data isn't the hard part. Understanding each other is. Ontology. Lineage. Semantic layers. Vector databases. I've been in data for over 15 years, and sometimes even I feel like I'm decoding a foreign language. We've turned simple ideas into jargon that makes non-data people tune out. Here's what these terms actually mean and why they matter for AI: ▶️ Ontology A shared definition of your core business concepts and how they relate. It gives AI clear concepts to reason about instead of guessing. ▶️ Entity A real world thing like a customer, product or event. It helps AI tell the difference between people, products and moments in time. ▶️ Metadata Data that explains other data. It tells AI what something means, how fresh it is and whether it can be trusted. ▶️ Physical layer Where data is stored and processed. It shapes how fast, scalable and reliable AI workloads can be. ▶️ Logical layer How data is organised conceptually, not physically. It shields AI from raw technical mess. ▶️ Semantic layer A business friendly layer with agreed definitions and metrics. It stops humans and AI arguing over what a number actually means. ▶️ Schema The formal structure of what data exists and what type it is. It gives consistency so AI knows what to expect. ▶️ Data modelling How entities and their relationships are designed. It reduces confusion in how AI interprets data. ▶️ Data virtualisation Accessing data from many sources without copying it all. It lets AI work across systems seamlessly. ▶️ Vector database A database that searches by similarity, not exact matches. It enables richer retrieval and context for AI. ▶️ Data pipeline How data flows from creation to consumption. It keeps AI fed with timely and relevant inputs. ▶️ Orchestration Coordinating when and how pipelines run. It keeps jobs reliable and in the right order. ▶️ Data quality How accurate, complete and consistent the data is. It directly affects confidence in AI outputs. ▶️ Observability Seeing what data systems are doing and spotting issues early. It helps catch drift and weird behaviour before damage is done. ▶️ Data lineage Where data comes from, how it changes and where it’s used. It adds transparency and explainability to AI decisions. None of this is magic. But together, it’s the foundation AI stands on. What other terms would you add as essential? ♻️ Repost to help someone get their idea into action. 🔔 Follow Clare Kitching for insights on unlocking value with data & AI. 💎 Get more from me with my free newsletter here: https://lnkd.in/giQ3b6Fi

As I finish sketching my “AI in 2026” observations, this last one ties everything together: As autonomy scales, responsibility becomes harder to locate. Once AI systems act continuously, coordinate with other agents, transact economically, and operate across organizational and jurisdictional boundaries, responsibility no longer maps cleanly to a single prompt, model, or human decision. Actions emerge from interactions. Decisions unfold over time. Outcomes are shaped by systems, not moments. When an agent triggers a financial loss, teams want to know what happened, why it happened, and where intervention was possible. When behavior drifts gradually, leaders need visibility into how decisions are being shaped by memory, incentives, and prior actions. Static policies and post-hoc audits don’t provide that clarity. This is why adaptive governance is becoming a practical design requirement. You can already see signals across research and product ecosystems. Recent work on autonomous agent oversight emphasizes runtime monitoring, traceability of decision paths, and intervention mechanisms that operate while systems are active. Explainability is moving closer to behavior itself: which tools were invoked, which memories were retrieved, and which constraints influenced an action. Startups are converging on the same needs from the ground up: ⭐ AgentOps.ai focuses on observability for agentic systems, tracing execution and surfacing failure modes in production. ⭐ CrewAI emphasizes role clarity and structured collaboration to make multi-agent behavior legible. ⭐ Portal26 and similar efforts focus on policy enforcement and auditability at the system level rather than trust in individual components. ⭐ Credo AI addresses governance from the organizational layer, helping enterprises operationalize AI policy, risk management, and accountability across models and systems. Responsibility shifts toward runtime visibility and control. Organizations begin to define responsibility across various layers, including agent behavior, orchestration logic, memory and data access, economic constraints, and human oversight. Governance becomes something systems participate in. Escalation paths are designed in advance. Intervention points are explicit. Logs and traces are preserved with intent, not just for debugging. This reaches beyond engineering. Legal teams, risk functions, procurement, and insurance increasingly ask for evidence of control rather than assurances of intent. Accountability becomes something that can be inspected and tested. By 2026, responsibility becomes a first-order design constraint. The organizations that scale autonomy successfully will build systems that can explain themselves, surface risk early, and invite intervention when boundaries are approached. Governance becomes part of the architecture. This is where AI stops being experimental capability and becomes institutional infrastructure.

I’ve seen plenty of AI agents that look brilliant in a demo but quickly become a liability in production. The reason is rarely a lack of model intelligence. It is a lack of operational visibility. As we move from experimentation to scaled, production-grade systems, particularly in highly regulated environments, we shift from predictable software to probabilistic agents. In a controlled sandbox, these systems are impressive. But the real world is messy; it is full of ambiguous inputs, tool timeouts, and shifting data distributions. In production, agents don't just break; they drift. An agent might handle 90% of tasks perfectly, while the remaining 10% fall into a "grey zone" of partial success. Without the right infrastructure, these failures aren't just hard to fix, they are hard to even find. Most teams are still trying to manage 2026 AI with 2010 monitoring tools. They see the final output, but they can't see the chain of reasoning that produced it. They notice a drop in stakeholder confidence, but they can’t point to the specific prompt version or tool call that caused the shift. To me, observability is the "missing middle" of the AI stack. It is the difference between hoping your agent works and having the evidence to prove it. For any enterprise to move beyond pilots, tracing and evaluation cannot be an afterthought. They must be the foundation of a responsible AI strategy. This is exactly why the work being done by platforms like orq.ai is so relevant right now. By bridging the gap between experimentation and live operations, they are providing the visibility needed to move AI out of the sandbox and into the core of the business with actual confidence. Building an agent is the easy part. Building a system that you can actually trust to run and govern is the real challenge. #AIOps #AgenticAI (in collaboration with orq ai)

Five years working with opaque ML models across safety critical self-driving and LLM systems has convinced me that we can't have transparent, accountable, steerable AI without being able to peer inside that black box. That's why I'm so excited to share Anthropic's latest breakthrough in Scaled interpretability - a technique that has identified 10M+ meaningful features in our Claude Sonnet model! This is a big step towards understanding AI systems more deeply, enabling greater control & reliability, and providing a roadmap for the field to build on Check out the technical details in our research report: https://lnkd.in/gsfmesUk Here's an example of the Golden Gate Bridge feature we found in the model, and what happens if Claude's responses if this feature is forced to activate strongly: