Production LLM applications need more than uptime monitoring. This guide shows how to build an observability workflow for prompts, responses, tokens, latency, provider errors, tool calls, and user outcomes—then use those signals to estimate cost, find regressions, and decide when your monitoring assumptions need to be recalculated.
Overview
LLM observability is the practice of recording enough context around each model interaction to explain what happened, how long it took, what it cost, and whether the result was useful. Traditional application monitoring still matters, but an LLM request has additional variables: model selection, prompt length, generated output, retrieval context, tool calls, retries, safety filters, and user feedback.
A useful monitoring design connects four levels of telemetry:
- Request level: request ID, trace ID, route, tenant, model, status, timestamps, and environment.
- Generation level: prompt and completion token counts, finish reason, response status, model parameters, and estimated cost.
- Workflow level: retrieval steps, tool calls, retries, validation results, and downstream actions.
- Outcome level: user ratings, task completion, escalation, correction, groundedness checks, or other application-specific quality signals.
Do not treat one aggregate number as a complete health signal. A chatbot can have normal latency while producing poor answers. A RAG system can return valid model responses while retrieving irrelevant documents. An agent can show acceptable token usage while repeatedly calling a tool that fails. LLM monitoring should therefore combine operational metrics, quality indicators, and cost estimates.
For background on related implementation concerns, see the LLM tool calling guide, the guide to prompt versioning and change management, and the LLM evaluation guide.
How to estimate
Begin with a simple unit model rather than attempting to predict every provider invoice detail. Estimate the cost and operational load of one request, then multiply it by expected traffic.
Estimated requests per period = active users × requests per user × traffic growth factor
Estimated input tokens = requests × average input tokens
Estimated output tokens = requests × average output tokens
Estimated model cost = (input tokens ÷ billing unit × input rate) + (output tokens ÷ billing unit × output rate)
The billing unit and rates are provider-specific inputs. Keep them in configuration rather than hard-coding them into dashboards, because model pricing, model identifiers, and usage fields can change. If a request includes multiple model calls, calculate each call separately and sum the results. For a RAG workflow, include the model call used to rewrite a query, generate an answer, or rerank content when those calls are billed independently.
For operational monitoring, calculate latency percentiles rather than relying only on averages. Track time to first token for streaming experiences and total completion time for the full answer. A practical starting dashboard includes p50, p95, and p99 latency, but the right alert boundaries depend on your user-facing service level and baseline. Set initial thresholds from observed behavior, then tighten them after collecting enough normal traffic.
For each metric, separate dimensions that help explain changes: model, route, prompt version, region, provider, tenant, success status, and workflow type. Avoid creating an unbounded dashboard dimension from raw user text, document IDs, or arbitrary tool arguments. High-cardinality data belongs in traces or sampled logs, not necessarily in every time-series metric.
Core LLM telemetry schema
{
"trace_id": "stable-trace-identifier",
"request_id": "application-request-identifier",
"timestamp": "event-time",
"environment": "production",
"route": "assistant-answer",
"model": "configured-model-name",
"provider": "configured-provider",
"prompt_version": "version-or-commit-id",
"input_tokens": 0,
"output_tokens": 0,
"time_to_first_token_ms": 0,
"total_latency_ms": 0,
"status": "success",
"finish_reason": "provider-value",
"estimated_cost": 0,
"retry_count": 0,
"retrieval_count": 0,
"tool_calls": [],
"quality_signal": null,
"redaction_status": "applied"
}The exact usage fields differ across SDKs and providers, so normalize them into an internal schema at your application boundary. Preserve the original provider response separately when permitted and useful for debugging, but do not make your dashboards dependent on one provider's field names. An OpenAI-compatible integration can simplify application code, yet usage reporting, streaming events, errors, and tool-call details may still require provider-specific adapters. The OpenAI-compatible API tutorial covers this integration pattern.
Inputs and assumptions
Before setting alerts or presenting a cost forecast, document the inputs that drive the estimate. At minimum, record:
- Traffic: requests per hour or day, active users, expected peaks, and retry behavior.
- Token shape: average and high-percentile input and output tokens. Include system prompts, conversation history, retrieved context, tool schemas, and tool results where applicable.
- Model mix: model and provider for each workflow step, including fallback models.
- Execution pattern: streaming or non-streaming responses, parallel calls, sequential chains, retries, and agent loops.
- Quality measurement: explicit ratings, human review, automated checks, task completion, citation checks, or escalation rates.
- Privacy policy: which fields are redacted, hashed, sampled, encrypted, or excluded from logs.
Use distributions instead of averages when the workload is uneven. A long conversation or large retrieved context can make average token usage look harmless while a smaller group creates most of the cost and latency. Report at least average and a high percentile for input tokens, output tokens, total latency, and tool-call count.
Privacy-safe sampling is usually more useful than indiscriminate raw logging. Store metadata for every request when feasible, but sample prompt and response bodies, redact secrets and personal data before storage, and provide a way to disable content capture for sensitive routes or tenants. Log hashes, lengths, classifier results, or redacted excerpts when full content is not needed. Never use observability data as a reason to retain more user content than the application requires.
Metrics by application type
| Application | Operational metrics | Quality and cost metrics |
|---|---|---|
| Chatbot | Time to first token, total latency, error rate, timeout rate, conversation length | Input and output tokens, user rating, abandonment, escalation, repeat-question rate |
| RAG system | Retrieval latency, generation latency, empty-result rate, context size, citation parsing errors | Retrieval success, groundedness checks, citation coverage, tokens per answer, correction rate |
| Agent workflow | Step latency, tool error rate, retry count, loop count, timeout rate | Task completion, tool-call count, failed action rate, tokens per task, human handoff rate |
For an agent, trace the complete workflow and each child span: planning, retrieval, model generation, tool execution, validation, and final response. This makes it possible to distinguish a slow model from a slow database or a tool that is being called repeatedly. For RAG, retain document or chunk identifiers in a protected trace field so an evaluator can inspect retrieval behavior without exposing the full document to every operator.
Worked examples
Example 1: estimating monthly token usage
Assume a support assistant handles 40,000 requests in a planning period. Each request uses an average of 1,200 input tokens and 350 output tokens. The estimate is:
- Input tokens: 40,000 × 1,200 = 48,000,000
- Output tokens: 40,000 × 350 = 14,000,000
- Total tokens: 62,000,000
To estimate model cost, apply the current input and output rates from the selected provider to the two token totals separately. Do not multiply total tokens by one blended rate unless you have deliberately created and documented that simplification. Recalculate the estimate if the prompt grows, conversation history is retained for more turns, retrieval adds context, or a fallback model is used more often.
Example 2: diagnosing a latency regression
Suppose a release keeps average latency close to its previous value, but p95 latency rises. Break the trace into spans and compare the prompt version, model, retrieval step, tool calls, and retry count. If the increase appears only on requests with large contexts, the likely next action is to inspect context construction and token distributions. If it appears only after provider errors, inspect retry policy and backoff. If only agent workflows are affected, compare tool execution time and loop counts with the prior release.
Record the finding as a baseline rather than declaring a universal threshold. For example, a team might alert when p95 latency exceeds its established target for several consecutive evaluation windows, when provider errors exceed the normal baseline, or when tool failures rise materially. The duration and percentage should reflect the product's tolerance for delay and failure; use them as configurable starting points, not permanent rules.
Example 3: separating quality from availability
A chatbot can return a successful HTTP response and still fail the user's task. Add a quality signal such as a thumbs-up or thumbs-down event, a resolved-versus-escalated outcome, or a sampled human review label. Join that signal to the prompt version and model used for the request. When quality falls after a prompt change, compare the affected cohort with a prior version and run a controlled evaluation set. Prompt versioning and regression tests are more reliable than trying to infer quality from latency or status codes alone.
When to recalculate
Revisit your LLM observability model whenever an input changes. Schedule a review during each model migration, prompt release, provider change, traffic forecast, cost review, or production incident. Recalculate sooner when any of these signals appear:
- Input or output token percentiles move materially.
- Latency changes for a particular route, model, region, or prompt version.
- Retries, timeouts, provider errors, or tool failures increase.
- RAG context size, retrieval counts, or empty-result rates change.
- Agent workflows add steps, tools, or fallback paths.
- User feedback, task completion, or escalation rates decline.
- Pricing inputs or provider usage fields change.
- Privacy requirements change the fields that can be logged or retained.
Use a release checklist to make the review repeatable: confirm model and prompt versions are captured; verify token and latency fields with a test request; check that error and retry spans are linked; inspect redaction on representative logs; compare current baselines with the previous release; and update the cost assumptions used by dashboards and forecasts.
The goal of LLM monitoring is not to collect every possible field. It is to make important decisions explainable. Start with traces, normalized usage, latency percentiles, provider and tool failures, and one meaningful quality signal. Then add workflow-specific measurements for RAG or agents. Keep assumptions visible, sample sensitive content carefully, and recalculate whenever traffic, models, prompts, pricing inputs, or workflow behavior changes.
For deployment context, pair this workflow with the LLM API cloud deployment checklist and review prompt injection defenses as part of the same production operating process.
