Your agent returned 200 OK and did the wrong thing

A deterministic service that breaks gives you a stack trace. Somewhere in the log is a line number, an exception type, and a path back to the fault.

An AI system that breaks gives you a clean response. Every span completed, no errors were raised, latency was well inside its budget, and the status code was 200. The agent issued a refund against the wrong invoice, and nothing in your monitoring registered anything unusual, because from the infrastructure's point of view nothing unusual happened.

Agentic systems fail in ways that look like success: well-formed but incorrect outputs, unnecessary tool calls, and actions that are syntactically valid and semantically wrong. That single property is why AI observability is a separate discipline rather than conventional observability applied to a new kind of service, and it is why teams deploying agents frequently cannot say which component was responsible when the output was wrong.

Three things that get called the same thing

The vocabulary is muddled, usually by vendors selling one and describing it as all three.

Monitoring tracks metrics you decided to track in advance. Latency, error rate, token spend, request volume. It answers questions you knew to ask, it is necessary, and for AI systems it is close to useless on its own, because the failure you care about does not move any of those numbers.

Observability captures enough of the execution that you can reconstruct what happened and answer questions you did not anticipate. For AI this means the whole decision path: each model call, each tool invocation, the arguments passed, what came back, and what the system did next.

Evaluation scores whether the output was any good. This is the part conventional observability has no equivalent for, because in ordinary software a response that returns successfully is, definitionally, correct. In AI it is not, and quality is therefore a first-class signal that has to be captured alongside latency and cost rather than inferred from them.

You need all three and they are frequently confused. A dashboard showing latency and spend is monitoring, and a team that has one often believes it has observability. It does not, and it will discover this the first time an agent does something expensive and wrong.

The vocabulary worth learning

Three terms carry most of the weight, and they are borrowed from distributed tracing rather than invented.

A span is one operation. A single model call, one tool invocation, a retrieval, a block of custom logic. It records inputs, outputs, duration, and whatever metadata you attach: tokens consumed, cost, which model version, cache hit or miss.

A trace is the complete set of spans for one request, arranged in a tree that preserves parent-child relationships. It is the call stack for an AI system. A trace shows not only what happened but the order it happened in and which step triggered which.

A session, sometimes called a thread, groups related traces. A multi-turn conversation is one session containing many traces, and a long-running workflow is the same. This matters because a great many AI failures are not visible in a single turn, only in the sequence: the agent that gradually loses the thread, or contradicts something it established four turns ago.

The nesting is where the value concentrates. In a multi-agent system, spans nested under the correct parent give you a chain of custody: which agent called which tool, with what arguments, what the tool returned, and what the calling agent did with that result. Without the nesting you have a flat list of events and no way to attribute a failure to a component, which is exactly the position most teams running agents are in.

What to capture, concretely

The list is longer than for ordinary services, and each item earns its place by being something you will want during an investigation you have not had yet.

The full prompt actually sent. Not the template. The assembled prompt including retrieved context, conversation history, system instructions and tool definitions. Most prompt bugs are assembly bugs, and a template tells you nothing about what the model received.

Every tool call with its arguments and its response. Both halves. A tool that was called with wrong arguments and a tool that returned something unexpected produce identical downstream symptoms and require different fixes.

Tools considered but not called, where your framework exposes this. The interesting question after a failure is frequently why the agent did not do the obvious thing, and that is unanswerable from a record of what it did do.

Reasoning output, where the model produces it. With the caveat covered below about what it is worth.

Model version and parameters. Provider models change under a stable name. A trace that does not record which version produced an output cannot explain a regression that arrived without any deployment on your side.

Token counts and cost per span. Cost attribution at the request level tells you the total. Cost attribution at the span level tells you which step is expensive, which is the actionable form.

Retrieved context, if there is retrieval. The passages, not just the fact that retrieval occurred. Most failures blamed on generation are retrieval failures, and you cannot distinguish them after the fact without the passages.

The volume is substantial. Traces for a busy agent system are considerably larger than conventional application logs, and sampling is usually necessary. Sample by trace rather than by span, since a partial trace is close to useless, and keep failures at a higher rate than successes.

A failure taxonomy worth having

Investigations go faster when you can name the category before you start looking, and AI failures sort into a small number of shapes that recur across systems.

Assembly failures. The prompt sent was not the prompt intended. Retrieved context landed in the wrong place, conversation history was truncated at a bad boundary, a template variable was empty. These are the most common and the least interesting, and they are invisible unless you captured the assembled prompt rather than the template.

Retrieval failures. The right passage was never found, or was found and truncated at a boundary that removed its meaning. Frequently misdiagnosed as a generation problem, because the output is fluent and wrong and the retrieval step reported success.

Tool failures. Called with wrong arguments, or called correctly and returned something unexpected. Distinguishing these requires both halves of the call, and they need opposite fixes.

Decision failures. The model chose the wrong tool, or chose not to use one it should have. The hardest category to see, because the trace records what happened and this failure is about what did not.

Coordination failures. Specific to multi-agent. Work was decomposed badly, delegated to the wrong specialist, or the merge dropped something. Only visible with properly nested spans.

Genuine model failures. The prompt was right, the context was right, the tools worked, and the model still produced something wrong. This is the category everyone assumes they are facing and it is the smallest of the six.

That ordering is worth internalising, because teams debug in almost exactly the reverse order: they suspect the model first, change the prompt, change the model, and eventually discover a truncation bug. Checking assembly first costs minutes and resolves a large share of incidents.

Evaluation, online and offline

Observability tells you what happened. It does not tell you whether it was any good, and that gap is where the discipline actually lives.

Offline evaluation runs a fixed set of cases against a candidate system before it ships. This is the regression guard: it catches the change that broke something previously working. It has the property that makes prediction industrialise, a repeatable number you can compare across versions, and it has the limitation that it only tests what you thought to include.

Online evaluation scores live production traffic. Some proportion of real traces get graded, either by rules, by a model acting as judge, or by a human. This catches what the fixed set does not, which is everything real users do that you did not anticipate, and it is where the surprising failures surface.

The loop between them is the point. A failure found in production becomes a case in the offline suite, so the same failure cannot recur silently. A system with observability and no evaluation produces evidence nobody acts on. A system with offline evaluation and no production scoring optimises against a set that drifts further from reality every week. The two together compound; either alone decays.

On the judge: using a model to grade another model's output is cheap, scales, and correlates reasonably with human judgement on many tasks. It also inherits the judge's biases, favours outputs resembling its own style, and is unreliable exactly where the task is hardest. Treat judge scores as a screening mechanism that routes work to humans rather than as ground truth, and calibrate the judge against human labels periodically rather than assuming the correlation holds.

Multi-agent makes all of this compulsory

For a single agent with a couple of tools, careful logging gets you a long way. Beyond that it stops being optional.

The failure could be in decomposition, in delegation, in an individual agent, in the merge, or in the coordination between any of them. Without nested spans preserving which agent did what, you have a wrong answer and five candidate explanations, and no way to narrow them without reproducing the run.

Reproduction is its own difficulty, because these systems are not deterministic. The same input can produce a different path, so "run it again" is not a debugging strategy. Session replay, reconstructing the exact sequence from a recorded trace, is the substitute, and it only works if the trace captured enough to reconstruct from.

The practical rule: the point at which you add a second agent is the point at which you need tracing, not the point at which you should start considering it. Teams routinely reach multi-agent before observability and then spend weeks unable to explain behaviour they are already shipping.

Starting from nothing

If none of this exists yet, the order matters more than the tooling, and the useful version is smaller than it sounds.

Week one: capture the assembled prompt and the tool calls. Not a platform, just structured logging of what was actually sent and what came back at each step, with a request identifier tying them together. This alone resolves the assembly and tool categories above, which between them account for a large share of incidents.

Week two: thirty test cases in a file. Inputs with expected outputs, run manually before each change. Crude, unglamorous, and it catches the regression that a prompt edit introduced somewhere you were not looking. The perfect evaluation framework does not get built; the crude one pays for itself immediately.

Week three: sample and read. Twenty production traces a week, read by someone who knows what good looks like. This finds systematic problems that aggregate metrics hide, particularly whole categories of request that always fail in the same way and never generate a complaint.

Then instrument properly, against the open standard, and choose a platform. By this point you know what you actually need from one, which is a much better position than choosing first and discovering your requirements afterwards.

The failure pattern worth avoiding is inverting this: selecting a platform, instrumenting comprehensively, generating dashboards nobody reads, and never building the thirty-case file that would have caught the regression.

The standard, and why it matters

OpenTelemetry has become the vendor-neutral base, with semantic conventions specific to generative AI defining how model calls, tool invocations and token usage should be represented.

This is worth caring about for one unglamorous reason: instrumentation is the expensive part, and it is the part you do not want to redo. Instrumenting against an open standard means the trace data can be routed to whichever platform you choose, alongside your existing infrastructure observability, and moving platforms later costs a configuration change rather than a reinstrumentation project.

The platform market is consolidating and the platforms differ less than their marketing suggests. Instrument once, against the standard, and treat platform choice as reversible. Teams that instrumented against a proprietary SDK have discovered that it is not.

What it costs

Being honest about this, because observability is usually presented as pure upside.

Storage and volume. Full traces including prompts and retrieved context are large. A high-traffic agent system generates data at a rate that makes retention policy a real decision rather than a default.

Latency, if you get it wrong. Synchronous evaluation on the request path adds latency to every request. Online scoring should be asynchronous and sampled, and this is a mistake teams make once.

Privacy exposure. Traces contain the full prompt, which contains whatever the user typed and whatever was retrieved on their behalf. That is frequently more sensitive than the database it came from, and it is now in a second system, possibly in a different jurisdiction, with its own access controls and retention. This deserves the same scrutiny as the primary data store and rarely receives it.

The judge's cost. Scoring production traffic with a model means additional inference, and scoring everything is not affordable at volume. Sampling policy is a real design decision, and biasing the sample toward likely failures beats sampling uniformly.

What is unresolved

Whether reasoning traces mean anything. Capturing a model's stated reasoning feels like observability and may not be. There is evidence that the stated chain does not always correspond to the computation that produced the answer, which would make it a plausible narrative rather than a record. If so, debugging from it is debugging from a story about the process rather than the process, and nobody has established how far the correspondence holds.

How to evaluate multi-step outcomes. Scoring a single response is tractable. Scoring a workflow that took forty steps, where step nineteen was wrong but recoverable and step thirty-one was subtly wrong and not recovered, is not, and no accepted methodology exists. Most current practice scores the final output, which cannot distinguish a system that got there reliably from one that got there by luck.

Whether judge models can be trusted at scale. The correlation with human judgement is decent on average and the disagreements are not random. Judges tend to fail on the same cases the system under test fails on, since both share training data and biases, which means the failure modes correlate exactly where independence would matter most.

What the right unit of analysis is. Conventional observability settled on the request. AI observability has candidates in the span, the trace, the session and the task, and different failures are visible at different levels. The field has not converged, which is one reason tooling comparisons are hard.

The counter-argument

Some pushback the enthusiasm deserves.

Much of this is standard distributed tracing with new vocabulary. Spans, traces and sessions are borrowed concepts, the tooling is largely conventional observability with LLM-aware span types, and a team with mature observability practice is closer to solved than the discourse suggests. The new part is quality as a first-class signal, and that is a smaller delta than a new discipline implies.

There is also a real risk of measurement theatre. It is possible to instrument comprehensively, generate dashboards, run judge models continuously, and improve nothing, because the data is never acted on. Observability that does not close a loop is expensive logging, and the loop closing, converting production failures into test cases and blocking regressions, is the part that gets deferred.

And for simple applications the honest answer is that this is overhead. A single model call with a well-understood prompt does not need distributed tracing. The threshold is roughly where multi-step workflows, tool use, or non-determinism enter, and below it a log file and a spreadsheet of test cases will serve.

The short version

AI systems fail in a way ordinary software does not: they return well-formed, confident, incorrect output with a success status, so nothing in conventional monitoring registers a problem. Latency was fine, no errors were raised, every span completed, and the agent did the wrong thing.

Three practices get conflated. Monitoring tracks metrics chosen in advance and is nearly useless alone here, because AI failures move none of them. Observability captures the full execution path so you can reconstruct what happened and answer questions you did not anticipate. Evaluation scores whether the output was good, which conventional observability has no equivalent for, because in ordinary software a successful response is by definition correct.

The vocabulary is borrowed from distributed tracing. A span is one operation, a trace is the tree of spans for one request, a session groups related traces. Nesting matters most, because in multi-agent systems it provides the chain of custody that lets a failure be attributed to a component rather than to the system as a whole. Capture the assembled prompt rather than the template, both halves of every tool call, model version, per-span cost, and retrieved passages.

Offline evaluation guards against regressions and only tests what you thought of. Online evaluation scores production traffic and surfaces what you did not. The loop between them is where the value is: a production failure becomes an offline case so it cannot recur silently. Judge models are a screening mechanism rather than ground truth, and their failures correlate with the system's own.

The threshold worth remembering: the moment you add a second agent is the moment tracing becomes mandatory, not the moment to start considering it. Teams routinely ship multi-agent systems before they can explain single-agent behaviour, and then spend weeks unable to say which component produced a result they are already serving to users.

Common questions

What is AI observability? Capturing enough of an AI system's execution that you can reconstruct what happened and answer questions you did not anticipate, including every model call, tool invocation, retrieval and decision. It differs from conventional observability by treating output quality as a first-class signal alongside latency, cost and errors, because an AI system can complete every operation successfully and still be entirely wrong, which conventional instrumentation has no way to register.

What is the difference between monitoring and observability for AI? Monitoring tracks metrics you decided to track in advance: latency, error rate, token spend. Observability captures the full execution trace so you can investigate failures you did not predict. For AI the distinction is unusually sharp, because the failures that matter move none of the monitored metrics. A request that returns in 400 milliseconds with no errors and a completely wrong answer looks healthy on every monitoring dashboard.

What are traces, spans and sessions? A span is a single operation such as a model call, tool invocation or retrieval, recording its inputs, outputs, duration and metadata. A trace is the complete tree of spans for one request, preserving which step triggered which, and functions as the call stack for an AI system. A session groups related traces, such as the turns of a conversation or the steps of a long workflow, which matters because many failures are only visible across a sequence rather than in any single turn.

Why can I not just use my existing observability stack? You largely can for the infrastructure layer, and should. What it lacks is the quality signal. Conventional observability assumes a response that completes successfully is correct, which holds for deterministic services and does not hold here. You also need span types that understand model calls and tool invocations, and the ability to capture assembled prompts and retrieved context, which generic instrumentation does not know to record.

What should I capture in a trace? The full assembled prompt rather than the template, since most prompt bugs are assembly bugs. Every tool call with both its arguments and its response, because wrong arguments and unexpected returns produce identical symptoms and need different fixes. Model version and parameters, since providers change models under stable names. Token count and cost per span rather than per request. Retrieved passages where retrieval is involved. And where your framework exposes it, the tools the agent considered and did not call.

What is the difference between online and offline evaluation? Offline evaluation runs a fixed set of cases against a system before it ships, guarding against regressions and testing only what you thought to include. Online evaluation scores samples of live production traffic, surfacing the failures real users produce that no fixed set anticipated. The loop between them carries the value: a failure found in production becomes a case in the offline suite so it cannot recur silently. Either practice alone decays.

Can I use a model to evaluate another model's output? As a screening mechanism, yes. It is cheap, it scales, and it correlates reasonably with human judgement on many tasks. Treat the scores as routing rather than as ground truth: they identify what a human should look at. The failure mode worth knowing is that judge models tend to fail on the same cases the system under test fails on, since they share training data and biases, so their errors correlate exactly where independence would matter most. Calibrate against human labels periodically rather than assuming the correlation persists.

When do I actually need this? The threshold is roughly where multi-step workflows, tool use or non-determinism enter. A single model call with a well-understood prompt does not need distributed tracing, and a log file with a spreadsheet of test cases will serve. The moment you add a second agent it becomes mandatory rather than advisable, because the failure could be in decomposition, delegation, an individual agent, the merge or the coordination between them, and without nested spans you have a wrong answer and five candidate explanations with no way to narrow them.