Writing Evals Skill

---
name: writing-evals
description: Build evaluations for an LLM feature or prompt. Use when the user wants to test a prompt, measure output quality, catch regressions in an AI feature, or asks how to know whether a prompt change made things better.
---

# Writing Evals

A prompt without an eval is not engineered, it is guessed at. Formatting and wording changes swing quality more than intuition predicts, and without a fixed test set you cannot tell an improvement from a coincidence.

## Start from real failures

The best eval cases are outputs that were actually wrong. Before generating anything, collect: production failures, complaints, cases where a human had to intervene, and the outputs that made someone lose confidence in the feature.

Twenty cases drawn from real traffic are worth more than two hundred synthetic ones, because synthetic cases test the failure modes you imagined and real ones test the failure modes you have.

## Cover the categories deliberately

1. **Typical**: the common inputs. The bulk of the set.
2. **Regression**: every real failure, as a case. This is why the set exists.
3. **Boundaries**: empty, minimum, maximum, one item, many items, unusual formatting.
4. **Should decline**: inputs where the correct behavior is to ask, refuse, or return the uncertain value. A feature that never declines is a feature that fabricates.
5. **Adversarial**: input containing instructions or attempts to change the task. Expected behavior is that the task is still performed on the content.
6. **Out of scope**: what the feature should turn away.

Report the count per category. A set that is ninety percent typical cases will pass while the feature keeps failing in production.

## Grade as deterministically as the task allows

- **Exact match or a code check** for anything structured. Always prefer this. It is cheap, stable, and not arguable.
- **Property assertions** in code where the shape matters more than the wording: valid schema, required fields present, no entity outside the source, length in range, no forbidden phrase.
- **Model-graded**, only where quality is genuinely subjective. Write the rubric so two careful humans would grade an output identically, give it the criteria one at a time, and require a quoted excerpt as justification for each judgment. Then check the judge against human labels before trusting it: an unvalidated judge is a second unmeasured system.

## Define the bar before running

Decide in advance: the pass rate required overall, which categories must be perfect (usually safety and should-decline), and what result blocks a change from shipping. A threshold chosen after seeing the numbers is not a threshold.

Measure consistency, not just an average. Running each case several times and requiring all runs to pass reveals the instability that a single run hides, and instability is what users experience as unreliability.

## Iterate honestly

Change one thing at a time. Record the prompt version, the model and its settings, the date, and the full result. Re-run the whole set, not the cases you were working on.

Read the failing outputs themselves. The score tells you something changed; only the transcripts tell you what.

Re-run the set on every model upgrade. Prompt behavior does not transfer reliably across model versions, and a set that has not been re-run since the last upgrade is measuring history.