Reliable LLM applications are tested like software, not judged from a handful of impressive conversations. This guide gives you a reusable evaluation workflow: define a golden dataset, choose metrics and rubrics, compare automated and human review, and add prompt regression checks before releasing a model, prompt, retrieval, or tool change.
Overview
LLM evaluation is the process of checking whether an application produces answers that are correct, relevant, appropriately written, safe, and useful for its intended task. The goal is not to find one universal score. The goal is to make quality visible and changes measurable.
A practical evaluation suite normally combines four parts:
- Test cases: Representative inputs, expected behavior, and any reference answers or required facts.
- Metrics: Measurable signals such as exact match, classification accuracy, retrieval recall, response length, latency, or cost.
- Rubrics: Clear criteria for judging qualities that are difficult to measure with simple string comparisons.
- Regression checks: A repeatable comparison that shows whether a change improved, preserved, or damaged performance.
Keep the evaluation boundary clear. Test the complete application when you need to understand user experience, and test individual components when you need to diagnose a failure. For example, a RAG application may require separate checks for document retrieval, citation or grounding behavior, answer quality, and refusal behavior. A tool-using agent may need tests for argument validity, tool selection, error recovery, and final response quality.
Evaluation should also reflect the application’s risk. A casual brainstorming assistant can tolerate more variation than a system that summarizes records, routes support requests, or generates structured data for another service. Define unacceptable failures before selecting a score.
Checklist by scenario
For a new LLM application
- Write down the application’s intended task and its out-of-scope requests.
- Identify the qualities that matter most: factual correctness, instruction following, relevance, tone, completeness, format, safety, or another task-specific requirement.
- Collect representative examples from real or carefully constructed use cases. Include easy, typical, ambiguous, and difficult inputs.
- Record the expected behavior for each example. A reference answer is useful, but some cases are better described with required facts, forbidden claims, or an acceptable-answer range.
- Run the same cases against a fixed model, prompt, configuration, and input version.
- Review failures individually before trusting an aggregate score.
For a prompt change
- Save the previous prompt and configuration so the comparison has a known baseline.
- Run the full golden dataset rather than only the example that motivated the edit.
- Compare both average results and per-case changes. An improved average can hide a serious regression in a high-priority case.
- Check formatting, refusals, citations, tool calls, and other output contracts separately from prose quality.
- Inspect changes in response length, token usage, latency, and failure or retry rates if those affect the product.
- Record the decision, reviewer, date, prompt version, model version, and known trade-offs.
For a RAG or knowledge-grounded application
- Test whether the correct source passages are retrieved for each question.
- Check whether the answer is supported by the supplied context rather than merely plausible.
- Include questions with missing, conflicting, outdated, or irrelevant information.
- Verify that the application admits uncertainty or declines to answer when the required evidence is absent.
- Evaluate source references, quotations, and document identifiers when users rely on them.
- Repeat the tests after changing chunking, embeddings, filters, reranking, metadata, or retrieval limits.
For implementation guidance, pair this checklist with the RAG production blueprint. If your system produces structured responses, also test its schema and recovery behavior using the guidance in structured output prompts.
For a tool-using or agent workflow
- Define which tool should be selected for each scenario and which tools must not be used.
- Validate tool arguments against the schema, including missing, malformed, or unsafe values.
- Test timeouts, empty results, permission errors, duplicate calls, and partial failures.
- Check that the final answer accurately reflects what the tool returned and does not claim an action was completed when it was not.
- Test prompt-injection and untrusted-content cases separately from normal task accuracy.
The function calling and tool-use guide covers schemas and error handling, while the prompt injection testing checklist can help expand adversarial cases.
Golden dataset template
A golden dataset is a versioned collection of evaluation cases that represents the behavior your application must preserve. It does not need to be large to be useful, but it should cover the scenarios that matter. Start with a spreadsheet, JSON file, or test database and use a stable identifier for every case.
{
"case_id": "support-001",
"category": "billing",
"input": "How can I update the billing contact?",
"context": "Optional retrieved passages or tool fixtures",
"expected_behavior": "Give the documented steps and do not invent account-specific details.",
"required_facts": ["Explain where billing settings are located"],
"forbidden_behavior": ["Claim that the contact was changed"],
"priority": "high",
"reference_answer": "Optional answer for comparison",
"notes": "Include expected refusal or escalation behavior if relevant"
}
Version the dataset when its expected behavior changes. Separate changes to test data from changes to application code so reviewers can tell whether a score moved because the system improved or because the test became easier.
Metrics and evaluation methods
Use more than one method because each captures a different failure mode.
| Method | Useful for | Limitations |
|---|---|---|
| Exact match or structured comparison | Labels, fields, enums, IDs, and strict JSON contracts | Penalizes valid wording variation and may miss semantic errors |
| Reference-based similarity | Comparing summaries, translations, or answers with known references | A reference may be incomplete, and similar wording is not proof of correctness |
| Retrieval metrics | Checking whether relevant documents or passages were returned | Good retrieval does not guarantee a grounded final answer |
| LLM-as-judge | Scoring relevance, completeness, style, or pairwise preference at scale | Judges can be inconsistent, biased toward certain styles, or overlook factual errors |
| Human review | High-risk cases, ambiguous answers, and rubric calibration | Requires time, training, and consistent reviewer instructions |
| Operational checks | Latency, error rate, retries, token usage, and tool failures | Operational success does not establish answer quality |
A useful rubric uses a small, explicit scale. For correctness, for example: 0 means materially wrong or unsupported, 1 means partially correct with a meaningful omission, and 2 means correct and supported. Apply separate scores for relevance, tone, format, and safety when they affect the task. Define what earns each score with examples, and allow an “unable to judge” result when the case lacks enough information.
What to double-check
- Inputs: Confirm that test cases include realistic spelling, formatting, language, length, and ambiguity variation.
- References: Check that expected answers are accurate, current, and not overly prescriptive where multiple answers are acceptable.
- Sampling: Make sure rare but consequential cases are not hidden by a large number of easy examples.
- Judge prompts: Give automated judges the rubric, relevant context, and a required output format. Do not treat their scores as ground truth without spot checks.
- Reproducibility: Record model identifier, prompt version, system instructions, retrieval settings, tool fixtures, and other variables that can affect results.
- Thresholds: Set release thresholds by risk and priority. A single critical failure may require investigation even when the overall average passes.
- Security: Include malicious instructions, sensitive-data requests, and untrusted retrieved content where those inputs are possible.
For conversation-based products, evaluate complete flows rather than isolated turns. Check whether the assistant preserves relevant state, asks for missing information, escalates appropriately, and recovers after an error. The guide to LLM chatbot conversation flows provides useful categories for these tests.
Common mistakes
- Testing only happy paths: Normal examples reveal baseline quality, but edge cases reveal whether the system is dependable.
- Using one score: An aggregate metric can conceal failures in safety, formatting, retrieval, or high-priority categories.
- Changing the test and the system together: Keep dataset edits separate from prompt or model edits to preserve a meaningful comparison.
- Confusing fluency with correctness: A confident, well-written answer can still be unsupported or wrong.
- Relying entirely on a model judge: Calibrate automated scoring against human-reviewed cases and investigate disagreement.
- Ignoring non-answer behavior: Test refusals, clarification questions, escalation, and uncertainty—not just successful answers.
- Skipping production feedback: Synthetic tests are useful, but anonymized failure cases from real usage often expose missing categories.
When to revisit
Run the evaluation suite before releasing a prompt, model, retrieval configuration, tool schema, safety rule, or application workflow change. Revisit it when production monitoring reveals a new failure pattern, when the knowledge base changes materially, or when users begin asking for a new task.
During quarterly maintenance or seasonal planning cycles, review whether the golden dataset still represents current usage. Retire obsolete cases, add recent failures, rebalance categories, and recheck expected answers. If provider APIs, model behavior, evaluation tooling, or output constraints change, rerun a baseline before interpreting new scores.
Use this release checklist as the final action:
- Freeze the candidate prompt, model, dataset, and configuration.
- Run automated checks across the complete golden dataset.
- Review all critical failures and a sample of passing cases.
- Compare results with the previous version by category and priority.
- Run human review for ambiguous, high-risk, or judge-disagreement cases.
- Record the decision and known limitations.
- After release, capture meaningful failures as new versioned test cases.
A dependable LLM testing checklist is a maintained engineering asset. Keep it close to your prompt library and deployment workflow, and update it whenever the application’s inputs, tools, knowledge, or users change.