Pairwise A/B LLM Judge

You are comparing two AI responses to the same input. Decide which better satisfies the criterion. You must resist the known biases of AI judges: position (earlier ≠ better), length (longer ≠ better), and confidence (assertive ≠ correct).

**Criterion**: [THE SINGLE DIMENSION, e.g. "Which response more accurately answers the user's question using only the provided context?": one dimension per judge; run multiple judges for multiple dimensions]

**What matters for this criterion**: [2-4 CONCRETE MARKERS, e.g. "claims traceable to the context; the actual question answered; uncertainty acknowledged where the context is thin"]
**What does NOT matter**: [THE EXCLUSIONS, e.g. "style, formatting, length, hedging tone": the judge must not reward these]

<input>
[THE ORIGINAL INPUT/QUESTION]
</input>

<response_a>
[RESPONSE A]
</response_a>

<response_b>
[RESPONSE B]
</response_b>

Respond in exactly this format:
Analysis A: [2-4 sentences: how A performs on the criterion, citing specific spans]
Analysis B: [2-4 sentences, same standard, same depth; analyze B on its own merits, not relative to A]
Decisive difference: [the ONE difference that most affects the criterion]
Verdict: A | B | TIE

Rules: analyze both before deciding, verdict-first reasoning back-fills justification. TIE is a legitimate verdict when the decisive difference is trivial; do not manufacture a winner. Judge only the stated criterion: a response can be worse-written and still win on accuracy.

---
HARNESS NOTE (for you, not the judge): run every comparison TWICE with A and B swapped. Count a win only when both orders agree; disagreement = TIE. Position bias is systematic in LLM judges and this swap is the standard control. Randomize which real system is "A" across your dataset, and if your judge model shares a family with either system under test, expect self-preference bias. Use a third-family judge for close calls.