LLM Evaluation Guide: Create Test Sets, Rubrics, and Regression Checks for AI Apps

Overview

An LLM application should be evaluated as a complete system rather than as an isolated model. The prompt, retrieved context, tools, output parser, safety controls, fallback behavior, and user interface can all affect the result. A useful evaluation process therefore connects each test case to a task, an expected behavior, and a decision rule.

Start by defining what success means for the workflow. For a support assistant, success may include answering from approved material, acknowledging uncertainty, and avoiding unsupported claims. For structured extraction, success may require valid JSON, correct field values, and an explicit null when information is missing. For a RAG application, the answer may need to be relevant, grounded in the supplied context, and supported by accurate citations.

Keep the first test set small enough to inspect carefully. A practical starting point is a mixture of common requests, difficult examples, known failures, ambiguous inputs, and adversarial or boundary cases. Store the inputs, expected properties, application version, prompt version, model identifier, retrieved context where relevant, output, scores, and evaluator notes. This record turns an informal prompt experiment into an auditable LLM testing framework.

For related implementation practices, see the prompt template library and versioning workflow and the LLM observability guide.

Checklist by scenario

Chat and instruction-following applications

• Define the user goal and the acceptable answer type before writing a score.
• Include ordinary requests, multi-step instructions, incomplete requests, and requests that should be refused or redirected.
• Score instruction following separately from helpfulness. An answer can be fluent and still fail a required constraint.
• Check factual claims against a reference source when accuracy matters; do not treat confident wording as evidence.
• Test conversation history, corrections, follow-up questions, and context-window boundaries.

A simple chat rubric can use a 0–2 scale for each dimension:

Extraction and classification

• Create a reference answer with field-level labels, not only a complete example.
• Test missing fields, conflicting values, unusual formats, long inputs, and extra information that should be ignored.
• Validate output structure separately from semantic correctness. Valid JSON does not guarantee correct values.
• Measure exact match where appropriate, but use normalized or field-level comparisons when formatting may vary.
• Record false positives and false negatives for classification tasks so threshold changes are visible.

For extraction, a useful rubric asks whether the output is parseable, whether required fields are present, whether values match the source, and whether unsupported values were invented. If a field cannot be determined, define in advance whether the expected behavior is null, an empty value, or a clarification request.

RAG and grounded-answer workflows

• Save the retrieved passages with each test run; an answer cannot be evaluated fairly without knowing the context provided to the model.
• Separate retrieval quality from generation quality. A correct answer may be impossible when the relevant passage was not retrieved.
• Include questions answered by the corpus, questions outside the corpus, and questions with conflicting or outdated passages.
• Check whether citations support the specific claim they follow rather than merely appearing in the response.
• Require an appropriate uncertainty response when the context does not contain enough information.

Use the RAG prompt engineering guide alongside these tests when refining grounding instructions. Evaluate faithfulness to retrieved context separately from general answer quality.

Tool-calling and agent workflows

• Test whether the model selects the correct tool, supplies valid arguments, and avoids unnecessary calls.
• Simulate tool errors, timeouts, empty results, partial results, and permission failures.
• Check that the final response accurately reflects tool output and does not claim an action succeeded when it failed.
• Set explicit limits for retries, loops, tool count, and total execution time.

The tool-calling guide can help translate these cases into schema and error-handling tests.

What to double-check

Dataset design

Ensure the test set represents real traffic without copying sensitive data unnecessarily. Remove or mask personal, confidential, and secret values before storing prompts, retrieved documents, or outputs. Keep a small protected holdout set that is not used while tuning the prompt. Otherwise, a workflow may appear to improve simply because it has been optimized for familiar examples.

Label each case by scenario, difficulty, expected behavior, and risk. Risk-based labels help teams spend review time where an error would have the greatest consequence. Include a failure archive: every confirmed production issue should become a regression case after sensitive details are removed.

Metrics and decision rules

Choose metrics that reflect the task. Useful measures may include pass rate, field-level accuracy, parse success, citation support, refusal appropriateness, latency, token use, and tool-call success. Do not compress all dimensions into one score unless the weighting is understood and documented. A high average can conceal a serious failure category.

Define release gates before comparing versions. For example, a new prompt may need to improve the overall pass rate without reducing factuality below an agreed threshold. Track both the aggregate result and the slice results for language, customer type, document type, difficulty, or other meaningful categories.

Human and model-assisted judging

Human review is valuable for ambiguous cases and for checking whether a rubric reflects actual user needs. Model-assisted judging can expand coverage, but it should be treated as an evaluation aid rather than unquestioned truth. Give the judge a clear rubric, the original input, the application output, and only the reference material needed for the decision. Compare a sample of judge decisions with human decisions and investigate consistent disagreement.

For pairwise comparisons, randomize which answer is shown first when practical, and preserve the prompt, model settings, and judge instructions. A preference result is not the same as factual correctness; use pairwise testing to compare versions, then use task-specific checks to understand why one version won.

Common mistakes

• Testing only happy paths: Add ambiguity, missing information, malformed input, long context, and out-of-scope requests.
• Changing several variables at once: Record prompt, model, retrieval settings, tools, and parsing changes so a regression has an identifiable cause.
• Using one broad quality score: Break quality into observable dimensions and retain failure reasons.
• Evaluating only the final text: Inspect retrieved context, tool traces, structured outputs, latency, and error handling.
• Overfitting to the test set: Maintain a holdout set and refresh cases from real, reviewed failures.
• Ignoring nondeterminism: Repeat selected cases when outputs vary, and report a range or distribution instead of relying on one run.
• Keeping evaluation data indefinitely: Apply retention, access, and redaction rules appropriate to the data and workflow.

Regression testing should be connected to version control. Save a run identifier with every result and compare the candidate against a known baseline. When a case fails, inspect the difference rather than immediately changing the prompt. The failure may come from retrieval, parsing, a model update, a tool response, or an evaluator inconsistency.

When to revisit

Run the core offline suite before releasing a new prompt, model, retrieval configuration, tool schema, parser, or safety rule. Re-run targeted suites after fixing a defect, adding a workflow, changing a system prompt, or modifying the application’s input format. Production monitoring should feed reviewed failures back into the dataset, while protecting sensitive information.

Review the rubric before seasonal planning cycles and whenever workflows or tools change. A rubric that made sense for a simple question-answering feature may not cover a later agent workflow or a new structured output requirement. Also check judge-model instructions, supported evaluation-library syntax, model identifiers, and metric definitions when the evaluation tooling changes. Keep these dependencies explicit in the repository so a future run can be reproduced.

Use this final release checklist:

1. Confirm the test set covers common, difficult, risky, and recently failed cases.
2. Verify that expected outputs and scoring rules are specific enough for two reviewers to apply consistently.
3. Run the candidate and baseline with recorded versions, settings, context, and tool traces.
4. Compare overall metrics and important slices, not only the average score.
5. Review material failures manually and classify their causes.
6. Block or approve the release according to predefined thresholds.
7. Store the results, add confirmed failures to the regression set, and schedule the next review.

Consistent LLM evaluation is an operating habit: define expected behavior, test it with representative cases, investigate differences, and update the suite as the application changes. For deployment-related checks, pair this process with the cloud deployment checklist and the prompt change-management workflow.