LLM cost optimization is easier when it is treated as an engineering control rather than a last-minute billing exercise. This guide shows how to measure token usage, estimate unit economics, and reduce AI API costs with prompt caching, model routing, context trimming, batching, rate limits, and feature-level budgets—without assuming that the most expensive model is always the best choice.
Overview
The cost of an LLM application usually comes from more than the model’s advertised input and output rates. A realistic estimate should include prompt tokens, generated tokens, retries, tool calls, cached and uncached context, batch or asynchronous processing, and the number of requests generated by each product feature. Infrastructure, storage, monitoring, and support may also matter, but token spend is the starting point for most AI workflow decisions.
A useful cost-control process has four stages:
- Measure: record tokens, latency, model, feature, outcome, and retry behavior for every request.
- Estimate: convert usage assumptions into cost per request, cost per successful task, and projected monthly spend.
- Optimize: test caching, smaller models, shorter context, batching, and routing in a controlled order.
- Protect: enforce budgets, rate limits, alerts, and fallback behavior before traffic grows.
Cost reduction should not be measured by token spend alone. A cheaper request that produces an unusable answer may increase total cost through human review, retries, support work, or failed automation. Pair cost metrics with quality metrics and regression tests. The LLM evaluation guide can help you create a repeatable test set and rubric for these comparisons.
How to estimate
Start with a simple request-level model. Separate input and output usage because providers commonly bill these categories differently:
request cost = (input tokens ÷ billing unit × input rate) + (output tokens ÷ billing unit × output rate)
Then adjust the estimate for operational behavior:
expected cost per task = request cost × average attempts per task + tool-call and retrieval costs
monthly cost = expected cost per task × monthly tasks + fixed platform costs
Use the provider’s current pricing page and billing definitions for the input rate, output rate, billing unit, cache treatment, batch treatment, and any minimums. Do not copy prices into application code. Store them as versioned configuration so a pricing change can be reviewed and applied without changing business logic.
For a useful estimate, log at least these fields:
- Feature or endpoint name
- Model and provider
- Input, output, cached, and uncached token counts when available
- Request status, retry count, and fallback model
- Latency and timeout information
- Quality result, such as evaluator score, validation status, or human correction
Calculate three values separately: cost per request, cost per successful result, and cost per active customer or internal user. The second value is often the most revealing. If a smaller model costs less per call but requires more retries, its apparent saving may disappear.
Inputs and assumptions
Before changing a model or prompt, create a small cost-model worksheet. Use a separate row for each feature because a summarizer, retrieval assistant, classifier, and agent can have very different token patterns.
| Input | What to record | Why it matters |
|---|---|---|
| Monthly tasks | Expected and peak task volume | Prevents average traffic from hiding capacity or budget risk |
| Input tokens | Typical, high-percentile, and maximum context | Long documents and chat history can dominate spend |
| Output tokens | Typical and maximum generated length | Unbounded responses create avoidable cost and latency |
| Attempts per task | Retries, validation failures, and fallbacks | One user action may create multiple billable calls |
| Model mix | Share of traffic by model | Routing changes the weighted average cost |
| Cache rate | Eligible requests and successful cache reuse | Cache value depends on stable, repeated prefixes or results |
Set a token budget by feature, not only by account. For example, a document-ingestion workflow may need a larger per-task allowance than a short classification endpoint. Add a maximum output-token setting, maximum context length, timeout, and retry limit to each route. A budget should define what happens when it is exceeded: reject the request, shorten the context, switch to a fallback model, queue it for later, or ask the user to narrow the task.
Choose optimizations by workload
| Technique | Best fit | Main trade-off |
|---|---|---|
| Prompt or response caching | Repeated instructions, stable reference text, duplicate requests | Requires safe cache keys, expiration, and invalidation rules |
| Smaller model | Classification, extraction, routing, and well-defined transformations | May reduce quality on ambiguous or complex tasks |
| Model routing | Mixed workloads with clear complexity differences | Needs a reliable classifier and fallback path |
| Batch processing | Non-urgent jobs such as indexing or scheduled summaries | Introduces delay and requires queue management |
| Request compression | Large prompts, repeated history, verbose retrieved passages | Compression can remove details needed for accuracy |
Worked examples
Example 1: Reducing context before changing models
Suppose a support assistant sends a system instruction, the complete conversation history, retrieved documents, and a response request on every turn. Its cost worksheet shows that retrieved text and old conversation turns make up most of the input tokens, while the generated answer is relatively short.
Test context trimming first. Keep the current user request, the instructions required for the task, and only the retrieved passages that pass a relevance threshold. Summarize older turns into a compact state object rather than sending the full transcript. Compare the original and trimmed versions on the same evaluation set. Record input tokens, answer quality, citation or grounding checks, latency, and cost per successful answer.
If quality remains within the team’s accepted range, the saving applies to every request on that route. If quality drops, restore the missing context selectively instead of abandoning trimming altogether.
Example 2: Routing simple work to a smaller model
Consider an application with two workloads: structured category classification and open-ended policy analysis. Route classification requests through a smaller, lower-cost model when the input passes schema and length checks. Send policy analysis to the stronger model, with a fallback if the classifier is uncertain or the output fails validation.
Estimate the blended cost with:
blended cost = (simple-work share × simple-route cost) + (complex-work share × complex-route cost)
Do not approve the route based on cost alone. Compare invalid-output rate, retry rate, evaluator scores, and human correction time. If the simpler route causes additional attempts, include those attempts in the cost per successful result.
Example 3: Caching repeated work
A document-processing pipeline may receive duplicate uploads or repeat the same extraction request during testing. A response cache can avoid recomputing identical, deterministic results. A prompt cache may help when a large instruction or reference prefix is reused, depending on the provider’s current support and billing rules.
Build the cache key from the model, prompt version, relevant input content, tool or schema version, and parameters that affect the result. Exclude sensitive data from shared caches unless the access model is explicit. Define an expiration period and invalidate entries when the prompt template, extraction schema, source document, or model behavior changes. Caching should improve cost without serving stale or cross-user results.
When to recalculate
Recalculate the worksheet whenever a provider changes pricing, billing units, cache support, batch rules, model limits, or API behavior. Also recalculate when traffic changes, a new feature launches, a prompt or retrieval strategy changes, or a model migration is planned. These events can alter both token volume and quality-related retries.
A practical operating rhythm is a monthly cost review plus an event-driven review before launches. Compare forecast with actual spend by feature, model, and environment. Investigate the largest variance rather than applying a blanket reduction. Check whether growth came from more users, longer context, higher output limits, retries, unexpected tool calls, or a change in routing.
Use this action checklist:
- Export the latest token and request data by feature.
- Update provider pricing and billing assumptions from current documentation.
- Recalculate typical, high-percentile, and peak monthly spend.
- Review cache hit rate, retry rate, fallback rate, and cost per successful task.
- Run quality tests before approving prompt compression or model changes.
- Set per-feature alerts, daily limits, and a documented response to budget breaches.
- Record the assumptions, pricing date, prompt version, and approved optimization.
Keep the worksheet, dashboards, and prompt versions connected. Observability is most useful when token usage can be traced to a specific feature and release; see the LLM observability guide for a production monitoring workflow. For prompt changes, use versioned templates and controlled comparisons as described in the prompt template library guide.
The goal is not to minimize every individual request. It is to make spend predictable while preserving the quality, reliability, and safety that the feature requires.