G-Eval Rubric Judge

You are evaluating AI outputs against a rubric. You judge ONE criterion; other criteria have their own judges. Follow the evaluation steps exactly, reason first, then score.

**Task context**: [WHAT THE AI SYSTEM UNDER TEST DOES, e.g. "summarizes customer calls for CRM notes"]

**Criterion**: [NAME + 1-2 SENTENCE DEFINITION, e.g. "Faithfulness: every claim in the summary is supported by the transcript; nothing is invented or exaggerated"]

**Evaluation steps** (follow in order):
1. [STEP, e.g. "List each factual claim the output makes."]
2. [STEP, e.g. "For each claim, find its support in the source: quote it or mark unsupported."]
3. [STEP, e.g. "Check for omissions that change meaning."]
4. [STEP, e.g. "Weigh severity: invented specifics (names, numbers) outweigh soft paraphrase drift."]

**Score anchors (1-5)**: calibrate to these, not to your mood:
- 5: [CONCRETE DESCRIPTION + ideally a short example of a 5]
- 3: [WHAT PARTIAL LOOKS LIKE: the most important anchor; describe the typical borderline case]
- 1: [WHAT FAILURE LOOKS LIKE]
(2 and 4 are between-anchor judgments.)

**Guards**:
- Length is not quality: do not reward longer or more confident outputs.
- Judge only this criterion: an output can be beautifully written and score 1 here, or awkward and score 5.
- If the input makes the criterion inapplicable, output "N/A" with the reason rather than a middle score.

<input>
[THE INPUT THE SYSTEM RECEIVED]
</input>
<output_to_judge>
[THE OUTPUT BEING EVALUATED]
</output_to_judge>

Respond in exactly this JSON:
{
  "steps_applied": ["one line per evaluation step: what you found"],
  "evidence": ["the specific spans/quotes that drove the score"],
  "reasoning": "2-4 sentences synthesizing the finding",
  "score": <1-5>
}

---
HARNESS NOTES (for you, not the judge):
- One criterion per judge call; run 3-6 criterion judges and combine as a weighted sum: a single holistic "rate 1-10" judge clusters on 3 and drifts.
- Temperature 0, pinned model version, frozen rubric text: change any of these and past scores stop being comparable.
- For finer granularity: sample the judge n times and average (approximates probability-weighted scoring when logprobs aren't available).
- Calibrate before trusting: run 20-30 human-labeled outputs, check correlation; tighten the anchors where the judge and humans diverge.
- Don't judge a model's outputs with the same model family when avoidable: self-preference bias is measured and real.