Eval Dataset Generator

Build an evaluation dataset for this LLM feature.

What the feature does: [THE TASK, THE INPUT, AND THE OUTPUT]
The prompt in use today:
"""
[PASTE THE CURRENT PROMPT]
"""
Real inputs and failures I have collected:
"""
[PASTE REAL EXAMPLES, ESPECIALLY THE ONES THAT WENT WRONG. THESE ARE WORTH MORE THAN ANY GENERATED CASE.]
"""
What counts as a correct output: [YOUR SUCCESS CRITERIA]
What a bad output costs: [THE CONSEQUENCE, so severity can be assigned]

Produce a dataset of [30-50] cases as a table with: id, category, input, expected output or expected property, grading method, and severity.

Cover these categories deliberately, and report the count in each:
1. **Typical cases**, drawn from my real examples. The bulk of the set.
2. **Known failures**: every real failure I gave you, turned into a case. These are the regression tests and the reason the set exists.
3. **Boundaries**: empty input, minimum and maximum length, a single item, the maximum number of items, unusual but valid formatting.
4. **Ambiguous cases** where the correct behavior is to ask, refuse, or return the uncertain value rather than produce a confident answer. A feature that never declines is a feature that hallucinates.
5. **Adversarial cases**: input that contains instructions, contradictory content, or an attempt to change the task. The expected output is that the task is still performed on the content.
6. **Out of scope**: inputs the feature should decline.

For each case, assign a grading method and say why:
- **Exact match** or a deterministic check, for structured output. Prefer this wherever it is possible; it is cheap, stable, and unarguable.
- **Property check** written as code, for outputs where the shape matters more than the wording (valid JSON, every required field present, no invented entity, length within range).
- **Model-graded**, only where quality is genuinely subjective. For these, supply the rubric with the specific pass and fail criteria, written so two careful humans would grade the same output identically.

Then produce:
1. **Coverage gaps**: what this set does not test, and what you would need from me to cover it.
2. **The baseline procedure**: how to run the set, what to record, and what pass rate on which categories should block a change from shipping.

Rules: mark every generated case as SYNTHETIC and every case drawn from my real data as REAL. Do not invent a case for a failure mode I have not observed and cannot describe; label those as hypothetical if you include them.