LLM evaluation turns subjective impressions into repeatable evidence. This guide shows how to build an evaluation dataset, choose metrics, estimate quality and operating cost, compare models, and track regressions after changes to prompts, retrieval, tools, or deployment settings.
Overview
An AI application can produce fluent answers and still fail in important ways: it may retrieve the wrong context, invent a detail, expose sensitive information, return invalid JSON, or take too long to respond. LLM evaluation is the discipline of testing those behaviors against defined expectations before and after a change.
The most useful evaluation workflow combines three layers:
- Automated checks: exact matches, schema validation, citation checks, latency measurements, token counts, and cost calculations.
- Rubric-based scoring: structured judgments for qualities such as relevance, correctness, clarity, faithfulness, and safety.
- Human review: targeted inspection of disagreements, failures, edge cases, and samples from production.
Do not reduce an AI app to one score. A weighted score can help compare versions, but a severe safety or privacy failure should remain a release blocker even when the average score improves. Treat evaluation as a decision system: define what must be true, measure it consistently, and record why a version passed or failed.
For retrieval-augmented applications, evaluate both retrieval and generation. The RAG prompt engineering guide provides useful context for designing grounded answers, while the LLM app with RAG guide connects architecture, prompting, and evaluation.
How to estimate
Start with a test set that represents the work your application is expected to perform. Each test case should contain an input, the relevant context or tool state, the expected behavior, and any constraints that matter. A simple record can use this structure:
{
"id": "support-014",
"input": "Summarize the refund terms for an annual plan.",
"context": ["approved_policy_document"],
"expected": "Answer only from the supplied policy and mention the applicable time window.",
"checks": ["grounded", "complete", "safe", "under_150_words"]
}Next, select metrics that correspond to user and business risks:
| Dimension | What to measure | Useful method |
|---|---|---|
| Correctness | Whether the answer reaches the expected conclusion | Reference answer, exact check, or rubric |
| Relevance | Whether the response addresses the request without unnecessary content | Pass/fail or a defined rating scale |
| Faithfulness | Whether claims are supported by supplied context | Claim review, citation validation, or rubric judge |
| Safety | Whether prohibited, unsafe, or sensitive behavior is avoided | Adversarial cases and hard release gates |
| Format | Whether output follows the required schema or structure | Parser, JSON Schema, or field-level checks |
| Performance | Response time and failure rate | Latency percentiles, timeout counts, and error logs |
| Cost | Estimated input and output consumption per request | Usage records multiplied by current provider rates |
For a rating-based metric, calculate the average across test cases:
Average score = total points awarded ÷ total possible points
For a weighted score, assign weights that reflect your application:
Overall score = (correctness × 0.35) + (faithfulness × 0.30) + (relevance × 0.20) + (format × 0.15)
The weights above are an example, not a universal standard. A medical, financial, or internal knowledge application may need stronger gates for safety, privacy, or evidence. Keep the weights documented so that comparisons remain meaningful.
Estimate regression impact by comparing the candidate version with a baseline on the same cases:
Metric change = candidate score − baseline score
A practical release rule might require no critical test failures, no meaningful drop in a protected metric, and an acceptable change in latency or cost. Define “meaningful” before reviewing results; otherwise teams tend to move the goalposts after seeing the output.
Inputs and assumptions
A reliable evaluation depends less on a large spreadsheet than on clear assumptions. Record the following inputs for every evaluation run:
- Application version: Include the prompt version, model identifier, retrieval configuration, tool definitions, parser version, and relevant feature flags.
- Dataset composition: Separate ordinary cases from edge cases, adversarial prompts, long inputs, multilingual requests, missing-context cases, and malformed requests.
- Scoring method: State whether each metric uses deterministic code, a human rubric, an evaluator model, or a combination.
- Thresholds: Define pass marks and release blockers at the metric or test-case level.
- Sampling assumptions: Explain how many cases are used, how they were selected, and whether they are fixed, rotating, or drawn from anonymized production traffic.
- Operational inputs: Capture input tokens, output tokens, retries, tool calls, latency, errors, and any current pricing values used for cost estimates.
Keep a small, permanent “golden set” for fast prompt regression testing. Add a larger rotating set for broader coverage. The golden set should include the cases most likely to reveal a breaking change: a known factual answer, a difficult retrieval query, a formatting constraint, an injection attempt, and a request that should be refused or redirected.
Evaluator models can speed up review, but their judgments are not automatically objective. Give the evaluator a narrow rubric, require structured output, and manually inspect a sample of its decisions. For important metrics, compare evaluator results with human judgments and track disagreement rather than hiding it.
For structured outputs, validate the result with a parser instead of asking an evaluator to decide whether the JSON “looks correct.” The structured output prompting guide covers schemas, function calling, and parsing reliability. You can also use developer utilities for repeatable JSON, SQL, regex, and related checks.
Worked examples
Example 1: Comparing two prompt versions
Suppose a support assistant is tested on 40 cases. Version A passes 34 correctness checks, 37 relevance checks, and all required format checks. Version B passes 36 correctness checks, 35 relevance checks, and all format checks.
The raw pass rates are:
- Version A: correctness 85%, relevance 92.5%, format 100%.
- Version B: correctness 90%, relevance 87.5%, format 100%.
If correctness has a 60% weight, relevance 25%, and format 15%, Version B may be preferred despite its relevance decline. However, inspect the five relevance failures before releasing it. A shorter response may score as less relevant under the rubric even if users prefer it, or the prompt may have removed useful explanations. The calculation identifies a trade-off; it does not replace review.
Example 2: Estimating request cost
Record the average input units and output units for a representative test set, then apply the current provider rates in your own configuration. The generic calculation is:
Estimated request cost = (input units × input rate) + (output units × output rate) + tool or retrieval costs
For a monthly estimate:
Estimated monthly cost = request cost × expected monthly requests × retry multiplier
Use a retry multiplier above one only when retries are part of the real workflow. Include separate scenarios for normal, heavy, and failure-prone traffic. Since rates and model configurations can change, store the date and assumptions beside the estimate rather than treating it as a permanent fact.
Example 3: RAG regression
A retrieval change improves answer relevance but causes the assistant to cite documents that do not support its claims. Keep retrieval recall, faithfulness, citation validity, and latency as separate metrics. A release can pass relevance while failing faithfulness, which should block deployment if unsupported claims are unacceptable for the use case.
For RAG systems, test empty retrieval results, conflicting documents, stale documents, duplicated passages, and malicious instructions embedded in source content. The prompt injection prevention checklist can help turn these risks into explicit test cases.
When to recalculate
Run the full evaluation suite whenever you change the model, system prompt, few-shot examples, retrieval settings, chunking, embedding configuration, tool schema, output parser, safety filter, or routing logic. A smaller regression suite should run on each pull request or prompt change when practical.
Recalculate operational estimates when provider pricing inputs, model limits, traffic forecasts, retry behavior, or average context size changes. Revisit the dataset when users introduce new intents, the product expands into another language or domain, or production logs reveal failure patterns missing from the test set.
Use this repeatable operating loop:
- Freeze the baseline version and dataset.
- Run the candidate against the same cases.
- Compare quality, safety, latency, and cost metrics.
- Inspect every critical failure and a sample of automated passes.
- Record the decision, assumptions, and follow-up cases.
- Promote representative production failures into the regression set.
Finally, connect evaluation results with observability. Log prompt and model versions, response metadata, token usage, latency, tool outcomes, and failure categories without storing sensitive content unnecessarily. The AI app observability guide offers a practical logging framework. A disciplined evaluation process is not a one-time benchmark: it is a living quality system that helps developers make safer, clearer decisions as an AI application changes.
