LLM observability: what to trace and evaluate in a live AI feature

LLM observability means recording what your AI feature actually does on every request: the prompt, the context it retrieved, the model call, the answer, the tokens it used, how long it took and how good the answer was. Without it, a wrong answer looks exactly like a right one in your logs, because both return a successful response.

This guide covers what to trace in each model call, the signals worth measuring, how to score answer quality in production, how to keep token cost and prompt logs under control, and how to choose tools. If your feature is already live and misbehaving, our AI integration services cover evaluation, cost and model upgrades for products in production.

What is LLM observability?

LLM observability is the practice of capturing traces, metrics and evaluation results for every large language model call, so a team can explain why an answer was wrong, slow or expensive. It extends normal application telemetry with the things only an AI feature has: prompts, retrieved context, token usage and answer quality.

Classic service health checks still matter. The Google SRE book names four golden signals for any service: latency, traffic, errors and saturation. An AI feature can pass all four and still fail its users. The model returns a confident, well-formatted answer that is out of date, made up or drawn from the wrong document, and nothing in your error rate moves.

That is the gap LLM observability fills. Traditional monitoring tells you a request failed; observability lets you open one request and see each step it took. For an AI feature, that view has to include the inputs and the quality of the output, not only the status code.

What to trace in every model call

Trace each user request as one trace, with a child span for every step: retrieval, each model call, each tool call and any post-processing. That structure lets you follow a bad answer back to the step that caused it, such as a retrieval that returned the wrong chunks.

You do not have to invent the field names. The OpenTelemetry GenAI semantic conventions define standard names for them. As of October 2026 they are maintained in their own repository, separate from the main OpenTelemetry semantic conventions, carry the status Development and have no tagged release yet. OpenTelemetry's versioning and stability rules say breaking changes may occur while a signal is in development, so pin the instrumentation version you use and expect renames.

The fields a small team should capture on each model-call span, using those conventions' names:

• What was called. The operation (gen_ai.operation.name, such as chat, embeddings, retrieval or execute_tool) and the model you asked for (gen_ai.request.model). The span name is the two together.
• What it used. Input and output tokens (gen_ai.usage.input_tokens and gen_ai.usage.output_tokens). The input count includes cached tokens.
• How it ended. The finish reasons (gen_ai.response.finish_reasons), which show when an answer was cut off by a length limit or a content filter, and error.type when the call failed.
• Your own context. The prompt template version, the feature name and an anonymous user or tenant ID, so you can group traces by what changed and who was affected.

The LLM observability signal map

Six signals cover the cost, speed, quality and safety of a live AI feature, and each one needs an alert that reaches a person. The table maps each signal to what to capture, why it matters and an example alert; set the thresholds from your own baseline traffic.

Time to first token is the delay before a streamed answer starts to appear, and it shapes how fast the feature feels. The OpenTelemetry conventions record it on streaming requests as time to first chunk, measured from when the client sends the request.

How to evaluate answer quality in production

Score a sample of live answers automatically, and send the low scorers to a person. Quality is the one signal a status code cannot show, so it needs its own measurement: an eval score attached to the trace it judged.

Start with an evaluation set built from real questions with agreed good answers, and run it before every prompt, model or retrieval change. In production, sample a share of live traffic and score it for the failures that matter to your feature, such as answers not supported by the retrieved chunks, off-topic answers or a missing citation. The conventions include a gen_ai.evaluation.result event that records the metric's name and score against the call it judged, so scores and traces stay together.

Automated judges, including a second model grading the first, are useful for volume but can be wrong in the same way the model is. Check a sample of their verdicts by hand. Then turn every bad trace you find into a new test case, so the same failure is caught before the next release. With no scores, there is nothing to turn into a test, and the same failure can ship again.

Tracking token cost and runaway usage

Track cost per request, per feature and per user, not only the monthly invoice. Token counts on every span make that possible, and a per-user view is how you spot one account or one bug burning through your budget.

The OWASP Top 10 for LLM Applications names this risk LLM10:2025 Unbounded Consumption. It describes Denial of Wallet attacks that exploit the cost-per-use model of hosted AI services, and its prevention list includes rate limits, per-user quotas and continuous monitoring of resource use. Observability supplies the data those controls need.

Two engineering bugs run up the same bill without any abuse: a retry loop that resends a long prompt, and an agent that keeps calling tools without a step limit. Both show up in traces as many spans for one request. Model changes move cost too, so keep an eye on provider shutdown dates such as the OpenAI API deprecations, because the replacement model may price tokens differently.

Logging prompts without leaking user data

Log metadata on every call, and log full prompts and answers only where you need them, with redaction, short retention and restricted access. Prompts often hold exactly the personal data your privacy policy promises to protect.

The OpenTelemetry conventions treat message content as opt-in: the attributes for input and output messages are marked Opt-In and carry a warning that they are likely to contain sensitive information, including user and personal data. That is the right default. The OWASP entry on sensitive information disclosure gives a related failure: a user receives a response containing another user's personal data.

A practical setup keeps token counts, timings, scores and IDs for every request, and stores full text only for sampled or flagged traces, after redacting emails, names and keys. Treat the trace store like a production database: limited access, an audit log and a deletion schedule. Before launch, penetration testing for startups should include the AI feature and the logs it writes.

Agent observability: tracing tools and multi-step runs

Agent observability means tracing a whole agent run, every planning step, model call and tool call, as one trace. Agents fail in ways a single model call cannot: they loop, pick the wrong tool or act on a bad intermediate answer.

The GenAI conventions include agent operations such as invoke_agent and execute_tool, and metrics that count the model and tool calls made in one agent invocation. Those counts are the first thing to alert on, because a run that suddenly makes far more tool calls than usual is either stuck or being misused.

For AI agent observability, also record which tool was called with which arguments, what it returned and whether the agent's next step used it. Pair that with hard limits on steps and spend per run, so a bad run stops on its own and leaves a trace you can read.

How to choose LLM observability tools

Choose LLM observability tools by how they collect data, where the data lives and how well they evaluate answers, not by feature lists. They fall into four broad groups, and a team can combine two.

• OpenTelemetry instrumentation into your existing backend. Libraries emit GenAI spans and metrics to the tracing and metrics stack you already run. Lowest lock-in; evaluation is yours to build.
• Dedicated LLM tracing and evaluation platforms. Built around prompts, traces, datasets and scoring. Some are open source and can be self-hosted, which matters when prompts hold customer data.
• AI features in general monitoring platforms. Useful when you already run one and want AI traces beside your other services.
• AI gateways and proxies. Sit between your app and the model provider, so they see every call and can add rate limits, caching and cost tracking in one place instead of in each service.

Ask every option the same questions. Does it accept OpenTelemetry data, so you can switch later? Can prompts stay in your region or your own cloud account? Can it score sampled live traffic and link each score to its trace? And how does its price grow with traces and stored text?

How we set up LLM observability

We instrument the AI feature you already run, then make the signals useful: one trace per request, token cost per user and feature, an evaluation set built from your own traffic and alerts that reach the right person. Our generative AI development services include the evaluation and cost work, and our DevOps consulting services cover the tracing pipeline, retention and on-call routing behind it.

When a model or runtime your feature depends on is near its end of life, our software end-of-life tracker lists the dates. The work is led by Thanos Gkanas, our Head of Engineering. Every engagement is under an NDA, and the IP is always yours.

Frequently asked questions