LLM observability

Tracing an LLM application. The useful thing to know here is a detail most people get wrong: the OpenTelemetry GenAI conventions exist, and they are not stable.

Verified 2026-08. This area moves; re-check before quoting.

Why standard telemetry isn’t enough

An LLM call produces a span with a duration and a status, like any HTTP call. What it doesn’t tell you: which prompt version, which model, how many tokens, what the tool call chain was, whether the answer was grounded, or what it cost.

An agent makes this worse — one user request becomes a tree of model calls, tool executions, retrievals and possibly sub-agents. Without a trace you cannot answer “why did it do that”.

OpenTelemetry GenAI conventions

The vendor-neutral standard. It defines span names, attribute keys, metric instruments and event names for model invocations, tool executions, agent runs, retrieval and memory operations.

with tracer.start_as_current_span("chat gpt-model") as span:
    span.set_attribute("gen_ai.system", "openai")
    span.set_attribute("gen_ai.request.model", MODEL_ID)
    span.set_attribute("gen_ai.request.temperature", 0.0)
    response = client.chat.completions.create(...)
    span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
    span.set_attribute("gen_ai.response.finish_reasons", ["stop"])

The status, precisely: as of 2026 the GenAI and MCP semantic conventions are still experimental / in development, with no public stabilisation date. In the v1.42.0 release (12 June 2026) all gen_ai.* attributes and spans moved out of the main semantic-conventions repository into a dedicated GenAI conventions repo — an organisational change giving fast-moving GenAI work its own release cadence, not a graduation to stable.

The practical consequence: attribute names can still change between releases. Adopt them — they’re the right direction and the alternative is proprietary lock-in — but pin your instrumentation version and expect to update. Saying “use OTel GenAI conventions” without that caveat is the common half-answer.

What to capture

Chunk IDs from retrieval are the underrated one. When an answer is wrong, knowing exactly which chunks were in context is the difference between debugging in minutes and guessing.

Cost per span lets you attribute spend to features, tenants and users, which is how you find the one workflow burning the budget.

Content logging

The tension: full prompts and responses are the most useful debugging data and the biggest privacy liability.

if should_sample(trace_id, rate=0.01) and not contains_pii(prompt):
    span.set_attribute("gen_ai.prompt", redact(prompt))

The workable policy: log metadata always, content on a small sample, redacted, with short retention. Never log content by default in a system handling personal data — traces are a PII store like any other. See ../14_guardrails_safety/03_pii_privacy_and_compliance.md.

The tooling

The argument for OTel plus your existing stack: an LLM call is one span in a request that also touched a database, a queue and three services. Keeping it in one trace beats correlating across two systems by timestamp.

The argument for a specialised tool: prompt playgrounds, eval integration, and UI built for reading conversation trees.

Both is normal — OTel for system-level correlation, a specialised tool for prompt iteration.

Dashboards worth having

• Cost: spend by feature, tenant and model; tokens per request over time.
• Latency: p50/p95 end to end, plus TTFT separately since it drives perceived speed.
• Quality: schema-validation pass rate, refusal rate, sampled groundedness score.
• Agent health: mean steps per run, step-limit-exhaustion rate, tool error rate.
• Traffic: query volume, length distribution, topic mix.

Step-limit exhaustion is a good leading indicator: a rising rate means agents are increasingly failing to complete, usually before users complain.

Interview angle

• “How do you trace an LLM application?” — OpenTelemetry with the GenAI semantic conventions, capturing model, tokens, cost, latency and finish reason per call, plus tool calls, retrieval chunk IDs and agent steps in the same trace as the rest of the request.
• “Are the OTel GenAI conventions stable?” — no. They remain experimental with no public stabilisation date, and moved to a dedicated repo in v1.42.0 for release cadence rather than as a graduation. Adopt them, pin your instrumentation version, and expect attribute names to change.
• “LangSmith or OpenTelemetry?” — usually both. OTel keeps the LLM call in the same trace as the database and queue spans it sits between; a specialised tool gives you prompt playgrounds and evaluation UI. Correlating two systems by timestamp is the thing to avoid.
• “What do you log that most people don’t?” — retrieved chunk IDs and per-span cost. Chunk IDs make a wrong answer debuggable; per-span cost lets you attribute spend to a feature or tenant.
• “Do you log prompts and responses?” — metadata always, content on a small redacted sample with short retention. Traces containing prompts are a PII store subject to the same rules as any other.
• “Which metric warns you an agent is degrading before users complain?” — step-limit exhaustion rate, and mean steps per run trending up. Both rise before task completion falls.