AI Observability
Capturing enough of a system's execution to reconstruct what happened, with quality as a first-class signal — because an AI system can complete every operation successfully and still be entirely wrong.
Reviewed July 16, 2026Stable
When not to use it
- For a single model call with a well-understood prompt, where a log file and a spreadsheet of test cases will serve.
- As a substitute for evaluation. Traces record what happened and say nothing about whether it was good.
Reach for something else instead
- Structured logging with a request identifier, which resolves most assembly and tool failures at a fraction of the effort.
- Conventional APM for the infrastructure layer, which it handles well and which lacks the quality signal.
Read more on the blog
- Your agent works in the demo because the demo is the easy caseA demo is not a small production system. It is the survivor of a search process designed to find something that works, which makes it evidence about your search rather than about the system.
- Most agent projects will be cancelled. This is why.Gartner puts the cancellation rate above 40% by the end of 2027. None of the three stated causes is a model problem, and the most common one is that the use case never needed an agent.
Sources & further reading
- OpenTelemetry, Semantic conventions for generative AI — the vendor-neutral instrumentation standard.
- Sigelman et al. (2010), Dapper, a Large-Scale Distributed Systems Tracing Infrastructure — where traces and spans come from.
- Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — how far model-as-judge scoring can be trusted.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Running evaluation synchronously on the request path, which adds latency to every request.
- Treating traces as private. They contain assembled prompts, which are frequently more sensitive than the database they drew from.
- Instrumenting comprehensively and never closing the loop, which is expensive logging rather than observability.
At a glance
FieldApplied AI
Unittrace, span, session
Distinguishing featurequality as a first-class signal
StandardOpenTelemetry GenAI conventions
Becomes mandatoryat the second agent
DifficultyPractical
Where this sits
A destination. 34 concepts lead here, and nothing in the corpus depends on it.
13Levelsteps in
34Needs firstconcepts
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works