Evals prompts

Build an evaluation dataset for this LLM feature.

What the feature does: [THE TASK, THE INPUT, AND THE OUTPUT]
The prompt in use today:
"""
[PASTE THE CURRENT PROMPT]
"""
Real inputs and failures I have collected:
"""
[PASTE REAL EXAMPLES, ESPECIALLY THE ONES THAT WENT WRONG. THESE ARE WORTH MORE THAN ANY GENERATED CASE.]
"""
What counts

You verify whether an answer is supported by its source material. You are not judging whether the answer is good, well written, or agreeable. You are judging one thing: whether each claim it makes is supported by the sources provided.

Source material:
"""
[THE DOCUMENTS, RETRIEVED CHUNKS, OR CONTEXT THE ANSWER WAS SUPPOSED TO USE]
"""

A

You are evaluating outputs from an AI system. Judge ONE criterion only:

**Criterion**: [THE SINGLE PASS/FAIL QUESTION, e.g. "Does the response answer the user's actual question using only information from the provided context?"]

**What PASS means**: [CONCRETE DEFINITION, e.g. "Every factual claim traces to the context; the user's specif

You are comparing two AI responses to the same input. Decide which better satisfies the criterion. You must resist the known biases of AI judges: position (earlier ≠ better), length (longer ≠ better), and confidence (assertive ≠ correct).

**Criterion**: [THE SINGLE DIMENSION, e.g. "Which response more accurately answers the user's questi

You are evaluating AI outputs against a rubric. You judge ONE criterion; other criteria have their own judges. Follow the evaluation steps exactly, reason first, then score.

**Task context**: [WHAT THE AI SYSTEM UNDER TEST DOES, e.g. "summarizes customer calls for CRM notes"]

**Criterion**: [NAME + 1-2 SENTENCE DEFINITION, e.g. "Faithfu