LLM Observability: What to Log in Production

Traditional application monitoring tells you that a request failed. LLM observability must tell you which step failed, what the model saw, what it decided, and whether the final result was still useful.

That distinction matters because a successful HTTP response can contain a bad answer. A model call can also look slow when the actual bottleneck was retrieval, a tool, or a retry hidden inside the agent loop.

This guide defines the minimum trace you need before an LLM feature reaches production. It extends the broader production LLM application checklist with an implementation-level telemetry design.

Start with a trace, not a log line

A trace represents one user-visible task. Each operation inside that task becomes a child span:

user_task
├── classify_intent
├── retrieve_documents
│   └── vector_search
├── generate_answer
└── validate_output

This hierarchy answers questions flat logs cannot:

• Did retrieval return irrelevant documents?
• Did the model ignore good evidence?
• Did the agent choose the wrong tool?
• Which retry doubled the cost?
• Where did the latency accumulate?

The tool does not matter as much as the trace shape. LangSmith, Langfuse, Arize Phoenix, OpenTelemetry, or an internal tracing system can all work if they preserve the task hierarchy and searchable metadata.

The minimum production trace schema

Use one stable schema across providers. If every model integration logs different fields, cross-model comparisons become manual work.

Do not make raw prompts your only source of truth. Log explicit version identifiers for the prompt, retrieval configuration, tool schemas, and policy bundle. A copied prompt does not reveal which code path assembled it.

Separate operational metrics from quality metrics

Operational metrics are deterministic and cheap:

• end-to-end latency and per-step latency;
• error and timeout rate;
• input and output tokens;
• estimated cost;
• retry count;
• tool-call count;
• cache hit rate;
• schema-validation failure rate.

Quality metrics require judgment:

• answer correctness;
• groundedness in supplied evidence;
• relevance to the request;
• appropriate tool selection;
• policy compliance;
• successful task completion.

Both belong on the same trace. Otherwise the team ends up comparing one dashboard full of latency data with a separate spreadsheet of manually reviewed outputs.

The existing guide to evaluating LLM outputs explains evaluator types. In production, run expensive semantic graders on a sample while keeping deterministic checks on every request.

Log enough context without creating a privacy incident

Full prompt capture is useful during debugging and dangerous when handled casually. Prompts may contain customer records, credentials, private documents, or regulated data.

Choose a capture policy per field:

1. Always retain: IDs, timings, versions, token counts, status codes, and aggregate scores.
2. Retain after redaction: user text, retrieved excerpts, tool arguments, and model outputs.
3. Do not retain: secrets, access tokens, raw credentials, and fields prohibited by customer or regulatory policy.
4. Sample selectively: complete traces for failures, low evaluator scores, or opted-in debugging sessions.

Hashing a value is not the same as anonymizing it. Stable hashes can still allow activity to be linked across requests. Document the threat model and retention window rather than assuming a telemetry vendor makes the decision for you.

Build dashboards around failure modes

A generic “LLM dashboard” usually becomes a wall of averages. Build views that answer operational questions instead.

Reliability dashboard

• successful tasks divided by attempted tasks;
• schema and tool failures;
• retries per successful task;
• human escalation rate;
• failure rate by prompt and model version.

Quality dashboard

• groundedness and correctness samples;
• user corrections;
• citation coverage;
• retrieval relevance;
• quality by intent or customer segment.

Cost and latency dashboard

• cost per successful task;
• P50, P95, and P99 latency;
• token use by workflow step;
• cache hit rate;
• expensive traces with poor quality scores.

Cost per request can reward a system that produces cheap failures. Cost per successful task connects the bill to the outcome, which is also the right lens for the LLM cost optimization strategies.

Create alerts that point to an action

Alert on symptoms that have an owner and a response:

• schema failures exceed the deployment baseline;
• tool timeouts increase for one dependency;
• retrieval relevance drops after an index update;
• cost per successful task rises after a model change;
• safety or prompt-injection detectors fire above a threshold;
• a new prompt version increases user corrections.

Avoid alerting on raw token volume without context. Higher volume may simply mean more users. Normalize metrics by tasks, users, or successful outcomes.

A practical rollout sequence

You do not need a complete observability platform on day one.

Stage 1: assign trace IDs and record model, prompt version, latency, tokens, errors, and final status.

Stage 2: add child spans for retrieval and tools. Record stable document and tool identifiers.

Stage 3: attach deterministic validators and user feedback to traces.

Stage 4: sample traces for semantic evaluation and human review.

Stage 5: promote repeated failure patterns into regression tests before the next release.

The final step closes the loop: production traces should improve the offline evaluation dataset, not disappear into a dashboard nobody reviews.

Production readiness checklist

• Every user task has one trace ID.
• Prompt, model, tool, and index versions are recorded.
• Retrieval and tool calls appear as child spans.
• Sensitive fields have explicit capture and retention policies.
• Operational and quality metrics share the same trace.
• Dashboards segment by version, intent, and tenant where appropriate.
• Alerts map to a documented owner and response.
• Failed and low-quality traces feed the regression dataset.

Sources and further reading

• LangSmith custom instrumentation documentation
• LangSmith observability overview
• LangSmith evaluation concepts