Lesson 3 - Pairwise Comparison & Preference

Welcome to Pairwise Comparison & Preference#

In Lesson 2 you handed the judge one Docent answer and a rubric and asked for a score – an absolute number like 4 out of 5. That works, but it leans on the model having a stable, well-calibrated internal scale, and models don’t really have one: ask the same judge to score the same answer on Monday and Friday and the number drifts. There is a more reliable question you can ask a judge, and it’s the one this lesson is about: given these two answers to the same question, which one is better?

That shift – from absolute scoring to relative comparison – matters because both humans and models are markedly more consistent at relative judgements than absolute ones. You may not be able to say whether a wine is “an 8” but you can reliably say you prefer this glass to that one. Pairwise comparison is the backbone of preference evaluation: it powers the human-preference datasets used to align models, and it’s exactly how public model leaderboards rank systems – collect thousands of “A vs B” verdicts and turn them into a win-rate. In this lesson you’ll build a real pairwise judge for Docent, our assistant for the Meridian serverless database docs, aggregate its verdicts into a win-rate, and then confront the failure mode that sinks naive pairwise judges: position bias, the judge’s tendency to prefer whichever answer it happens to see first.

In this lesson, you will:

• Understand why relative (“A is better than B”) judgements are more reliable than absolute scores, and how they aggregate into a win-rate
• Build a real judge_pairwise(question, answer_a, answer_b) on claude-haiku-4-5 that returns a structured winner and reason
• Run a good Docent variant against a vague one across many questions and compute a win-rate
• See position bias happen live – verdicts that flip when you swap the order – and fix it by running both orders and counting only order-consistent verdicts
• Recognize the other pairwise pitfalls: verbosity bias, ties, and broken transitivity

Why pairwise, and what a win-rate is#

Absolute scoring asks the judge to place an answer on a fixed scale it has to invent and hold steady. Pairwise judging asks a strictly easier question – which of these two is better? – and easier questions get more reliable answers. You give the judge the same question and two candidate answers, call them A and B, and it returns one of three verdicts: A wins, B wins, or tie, ideally with a one-line reason.

One comparison is just one data point. The power comes from aggregation. Run the same two systems – say, two versions of Docent – head-to-head across many questions, and count how often system A’s answer beats system B’s. That fraction is the win-rate:

A win-rate of 0.5 means the two systems are indistinguishable to the judge; 0.75 means A is preferred three times out of four. This single number is what leaderboards report, what you watch when you tweak a prompt, and the cleanest way to answer “did my change actually make Docent better?” – not in the abstract, but relative to the version I had before.

Building a pairwise judge for Docent#

We’ll compare two Docent variants on the same Meridian questions. Variant good answers crisply and correctly; variant worse is the kind of vague, padded answer a weaker prompt produces – fluent, on-topic, but light on the actual fact. To keep the lesson focused on judging (and the live calls tiny), these two answers are given as fixed strings, standing in for the outputs of two Docent configurations you’d normally generate.

The judge itself is one small claude-haiku-4-5 call. We ask for a strict JSON verdict and explicitly tell it not to reward length – a first, cheap defense against verbosity bias:

import warnings, json
warnings.filterwarnings("ignore")
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

# Four Meridian questions, each with a GOOD Docent answer and a WORSE one.
# The worse answer is vague/padded -- but the judge is never told which is which.
CASES = [
    {
        "q": "How many requests per minute does the Pro plan allow?",
        "good": "The Pro plan allows 600 requests per minute.",
        "worse": ("Great question! Rate limits depend on your plan tier, and the Pro plan "
                  "is quite generous compared to Free. You get a substantially higher "
                  "allowance of requests each minute, which should be plenty for most "
                  "production workloads and typical traffic patterns."),
    },
    {
        "q": "What does a 401 error mean?",
        "good": "A 401 error means the API key is missing or invalid.",
        "worse": ("A 401 is an HTTP status code in the 4xx family, which signals a "
                  "client-side problem with your request. It generally relates to your "
                  "credentials or permissions on the endpoint you are calling, so you "
                  "should review how your request is being authorized."),
    },
    {
        "q": "How long is the grace period before a deleted database is permanently purged?",
        "good": "The grace period is 24 hours; after that the data and backups are permanently purged.",
        "worse": ("When you delete a database it is soft-deleted first and can be restored "
                  "for a short window from the dashboard before it is purged for good. "
                  "The exact retention depends on your configuration, but you do have "
                  "some time to recover it if you act reasonably quickly."),
    },
    {
        "q": "How long are backups retained on the Scale plan?",
        "good": "On the Scale plan, backups are retained for 90 days.",
        "worse": ("Backup retention scales with your plan, and the Scale plan sits at the "
                  "top so it keeps backups the longest of any tier. This gives enterprise "
                  "teams a long recovery window well beyond what the cheaper plans offer "
                  "for their daily automatic backups."),
    },
]

def judge_pairwise(question, answer_a, answer_b, model="claude-haiku-4-5"):
    prompt = (
        "You compare two answers to a documentation question and decide which is better. "
        "Better means more accurate, directly responsive, and complete. Do NOT prefer an "
        "answer just because it is longer.\n\n"
        f"Question: {question}\n\n"
        f"Answer A:\n{answer_a}\n\n"
        f"Answer B:\n{answer_b}\n\n"
        'Respond with ONLY a JSON object: {"winner": "A" | "B" | "tie", "reason": "<one short sentence>"}'
    )
    msg = client.messages.create(model=model, max_tokens=120,
                                 messages=[{"role": "user", "content": prompt}])
    text = msg.content[0].text.strip()
    start, end = text.find("{"), text.rfind("}")
    return json.loads(text[start:end + 1])

# Compute the win-rate the RIGHT way: run each pair in BOTH orders and count only
# verdicts that agree. In order 1 the GOOD answer sits in slot A; in order 2 it sits
# in slot B. good_wins() maps the raw A/B verdict back to whether GOOD won.
def good_wins(verdict, good_is):
    w = verdict["winner"]
    if w == "tie":
        return "tie"
    return "good" if w == good_is else "worse"

consistent = 0
good_consistent_wins = 0
print(f"{'question':42s} {'order1(good=A)':14s} {'order2(good=B)':14s} verdict")
print("-" * 92)
for c in CASES:
    v1 = judge_pairwise(c["q"], c["good"], c["worse"])   # GOOD in position A
    v2 = judge_pairwise(c["q"], c["worse"], c["good"])   # GOOD in position B
    r1 = good_wins(v1, "A")
    r2 = good_wins(v2, "B")
    agree = (r1 == r2)
    tag = "CONSISTENT" if agree else "FLIPPED (position bias)"
    if agree:
        consistent += 1
        if r1 == "good":
            good_consistent_wins += 1
    print(f"{c['q'][:42]:42s} {r1:14s} {r2:14s} {tag}")

print("-" * 92)
print(f"order-consistent verdicts : {consistent}/{len(CASES)}")
if consistent:
    print(f"GOOD win-rate (consistent only) : {good_consistent_wins}/{consistent} "
          f"= {good_consistent_wins / consistent:.2f}")

Here is a real run against claude-haiku-4-5. Because it’s a live model, the exact verdicts can shift slightly on a re-run, but this run was stable across repeats:

question                                   order1(good=A) order2(good=B) verdict
--------------------------------------------------------------------------------------------
How many requests per minute does the Pro  good           good           CONSISTENT
What does a 401 error mean?                worse          worse          CONSISTENT
How long is the grace period before a dele good           good           CONSISTENT
How long are backups retained on the Scale good           good           CONSISTENT
--------------------------------------------------------------------------------------------
order-consistent verdicts : 4/4
GOOD win-rate (consistent only) : 3/4 = 0.75

The good variant wins three of four comparisons, for a win-rate of 0.75 – a clean, defensible “the good Docent is preferred three times out of four.” But look at the one it lost: the 401 question, where the crisp correct answer (“the API key is missing or invalid”) lost to the vague, padded one that never actually says what a 401 means. That loss is consistent across both orders, so it is not position bias – it’s the judge genuinely preferring the longer, more elaborate-sounding answer. That’s verbosity bias, and we’ll come back to it. First, the headline pitfall.

Position bias: the verdict that depends on the order#

Here is the single most important fact about pairwise judging: judges often prefer whichever answer they see first, regardless of quality. The verdict can depend on the order you presented the two answers in, not just their content. If you only ever run the comparison one way, you can’t tell a genuine preference apart from a positional reflex – and your win-rate is quietly corrupted.

The clean way to expose this is to hand the judge two answers of genuinely equal quality and force it to pick one (no ties allowed). An unbiased judge would have no basis to choose, so which text it picks should be stable when you swap the order. If instead it keeps choosing whatever sits in the first slot, the winning text will flip the moment you swap A and B – and that flip is position bias, caught red-handed.

import warnings, json
warnings.filterwarnings("ignore")
import anthropic
client = anthropic.Anthropic()

# Two EQUALLY GOOD paraphrase answers per question. We FORCE a pick (no tie). If the
# judge were unbiased the winning TEXT would be stable across orders; if it favours
# position, the winning text FLIPS when we swap the two answers.
PAIRS = [
    {"q": "How many requests per minute does the Pro plan allow?",
     "x": "The Pro plan allows 600 requests per minute.",
     "y": "Pro gives you 600 requests each minute."},
    {"q": "What does a 401 error mean?",
     "x": "A 401 means the API key is missing or invalid.",
     "y": "It means your API key is either absent or not valid."},
    {"q": "How long is the grace period before a deleted database is purged?",
     "x": "The grace period is 24 hours before permanent deletion.",
     "y": "You have 24 hours before the data is permanently removed."},
    {"q": "How long are backups retained on the Scale plan?",
     "x": "Scale keeps backups for 90 days.",
     "y": "On Scale, backup retention is 90 days."},
]

def judge_forced(question, answer_a, answer_b, model="claude-haiku-4-5"):
    prompt = ("Which answer is better? You MUST choose A or B. Ties are not allowed; "
              "if they seem equal, still pick the single best one.\n\n"
              f"Question: {question}\n\nAnswer A:\n{answer_a}\n\nAnswer B:\n{answer_b}\n\n"
              'Respond with ONLY JSON, reason under 10 words: {"winner":"A"|"B","reason":"<short>"}')
    msg = client.messages.create(model=model, max_tokens=150,
                                 messages=[{"role": "user", "content": prompt}])
    t = msg.content[0].text.strip()
    return json.loads(t[t.find("{"):t.rfind("}") + 1])

flips = 0
for c in PAIRS:
    v1 = judge_forced(c["q"], c["x"], c["y"])   # X in slot A
    v2 = judge_forced(c["q"], c["y"], c["x"])   # Y in slot A
    win1 = "x" if v1["winner"] == "A" else "y"  # which TEXT won, order 1
    win2 = "y" if v2["winner"] == "A" else "x"  # which TEXT won, order 2
    flip = win1 != win2                         # winner changed with the order
    flips += flip
    # a flip means the judge picked the SAME slot both times -> pure position bias
    note = f"FLIP: stuck on slot {v1['winner']} (position bias)" if flip else "consistent"
    print(f"{c['q'][:36]:36s} slotA-o1:{v1['winner']} slotA-o2:{v2['winner']} "
          f"winText:{win1}/{win2} -> {note}")
print("-" * 78)
print(f"order-inconsistent (flipped) verdicts : {flips}/{len(PAIRS)}")

A real run. The exact count varies from run to run – which is itself part of the point – but every run we tried showed at least one flip, and the backups pair flipped on every run:

How many requests per minute does th slotA-o1:A slotA-o2:B winText:x/x -> consistent
What does a 401 error mean?          slotA-o1:B slotA-o2:B winText:y/x -> FLIP: stuck on slot B (position bias)
How long is the grace period before  slotA-o1:A slotA-o2:B winText:x/x -> consistent
How long are backups retained on the slotA-o1:A slotA-o2:A winText:x/y -> FLIP: stuck on slot A (position bias)
------------------------------------------------------------------------------
order-inconsistent (flipped) verdicts : 2/4

Two of the four verdicts flipped just by swapping the order – and read how they flipped. On the backups question the judge picked slot A both times, so the winning text reversed from x to y; on the 401 question it picked slot B both times, reversing y to x. In each flip the judge locked onto a position and ignored which answer actually sat there. Re-running this experiment, the flip count bounces between one and two out of four and the 401 pair is a coin toss – but the backups pair flips every single time, a rock-solid, reproducible chunk of pure position bias. Run this once, one order only, and you’d walk away with four confident verdicts, at least one of which is noise, and you’d have no way to know which.

The fix: run both orders, count only what agrees#

You already saw the remedy in the first code block, and now you know why it’s there. Judge every pair in both orders – (A, B) and (B, A) – and only trust a verdict when the same answer wins both times. When the two orders agree, position wasn’t the deciding factor, so the verdict reflects quality. When they disagree, the comparison is order-dependent and you throw it out, counting it as a tie for win-rate purposes. That’s exactly why the good-vs-worse win-rate above was computed over order-consistent verdicts only: every number in a win-rate you report should have survived the swap. It doubles your judge calls, but it’s the difference between a win-rate that measures quality and one that measures seating position. (If you can afford more calls, running each order two or three times and taking the majority hardens it further against run-to-run noise.)

The other pitfalls: verbosity, ties, and transitivity#

Position bias is the loudest problem but not the only one.

Verbosity bias is the one you already saw bite: on the 401 question, the padded answer that never states the fact beat the crisp correct one, consistently in both orders. Judges tend to read longer and more elaborate as more thorough and higher quality, even when the extra words add nothing or actively dodge the question. The instruction “do not prefer an answer just because it is longer” helps but doesn’t cure it. Structural defenses work better: normalize answer length before comparing, or add an explicit rubric axis (“does it state the specific fact?”) so the judge grades substance, not volume. The next lesson on calibration measures exactly how far your judge’s verdicts drift from human ones for reasons like this.

Ties are real signal, not a nuisance to suppress. Two answers often are equally good, and a judge forced to break a tie (as we deliberately forced it above) is precisely where position bias takes over. Allow “tie” as a first-class verdict, exclude ties from the win-rate denominator, and treat an order-flipped verdict as a tie – because a verdict that depends on order is, by definition, not a real preference.

Transitivity breaks when you rank more than two systems. Pairwise only ever compares two things, so to rank systems A, B, and C you run the pairs and hope the results are consistent: if A beats B and B beats C, you’d like A to beat C. Judges don’t guarantee this – you can get the rock-paper-scissors cycle A > B > C > A. Real leaderboards handle it by collecting many pairwise comparisons and fitting a rating model (Elo or Bradley-Terry) that turns a pile of noisy, sometimes-cyclic verdicts into a single consistent ranking with a score per system. For comparing two Docent variants a plain win-rate is enough; once you’re ranking several, reach for a rating model rather than trusting a chain of individual pairwise calls.

Practice Exercises#

Exercise 1: Measure the position-bias rate on the good-vs-worse pairs#

The first code block counted order-consistent verdicts but didn’t report how many flipped. Modify the good-vs-worse loop to also count, per question, whether r1 != r2 (the winner changed when the order swapped), and print a “position-bias rate” = flipped / total. Run it and compare that rate to the one from the equal-answer experiment. Explain why clearly-different answers flip far less often than equal ones.

Exercise 2: Turn per-question verdicts into a reported win-rate with ties excluded#

Extend judge_pairwise’s aggregation into a function win_rate(cases) that runs both orders for every case, classifies each as good, worse, or tie (counting an order-flip as a tie), and returns good_wins / (good_wins + worse_wins) – ties excluded from the denominator, exactly as the formula in the lesson defines it. Confirm that on the four cases it reproduces the 0.75 (or close) you saw, and print how many cases were dropped as ties.

Exercise 3: Try to reduce verbosity bias on the 401 case#

The 401 comparison is where the padded answer beat the correct one. Attack it two ways and see which helps: (a) rewrite the judge prompt to include an explicit checklist – “Prefer the answer that states the specific fact the question asks for; ignore tone and length” – and re-run just that pair in both orders; (b) truncate the verbose answer to roughly the length of the good one before judging. Report whether the good answer now wins consistently under each intervention.

Summary#

Pairwise comparison asks a judge the easier, more reliable question – which of these two answers is better? – instead of demanding an absolute score, because both people and models are steadier at relative judgements than absolute ones. You aggregate many “A vs B” verdicts into a win-rate, the fraction of questions where one system’s answer beats another’s, which is how preference datasets are built and how leaderboards rank models. You built a real judge_pairwise on claude-haiku-4-5, ran a good Docent variant against a vague one, and got a 0.75 win-rate – with one telling loss to verbosity bias, where a padded answer that never stated the fact beat the crisp correct one in both orders. Then you exposed the headline pitfall, position bias: forcing the judge to choose between equal-quality answers, verdicts flipped just by swapping the order – one to two of four on any given run, with one pair flipping every run because the judge locked onto a fixed slot regardless of content. The fix is structural – run every pair in both orders and count only the verdicts that agree, treating order-flips as ties – which is exactly how the win-rate here was computed. You also met the remaining hazards: verbosity bias, ties as real signal, and broken transitivity when ranking more than two systems (where a rating model beats a chain of pairwise calls).

Key Concepts#

• Pairwise comparison – give the judge a question and two answers and ask which is better (A, B, or tie) with a reason; more reliable than absolute scoring because relative judgements are more stable.
• Win-rate – the fraction of judged questions where one system’s answer is preferred over another’s, ties excluded; the aggregate metric behind preference evaluation and leaderboards.
• Position bias – the judge’s tendency to prefer whichever answer it sees first; verdicts that depend on order are position bias, not quality.
• The two-order fix – judge every pair as (A, B) and (B, A) and trust only verdicts where the same answer wins both times; count order-flips as ties.
• Verbosity bias – judges over-reward longer, more elaborate answers even when the extra words dodge the question; fight it with length normalization and fact-focused rubric axes.
• Ties and transitivity – allow ties as a real verdict and exclude them from the win-rate; and remember pairwise verdicts can be non-transitive (A > B > C > A), so rank many systems with a rating model (Elo / Bradley-Terry), not a chain of comparisons.

Why This Matters#

Pairwise win-rate is the metric you’ll actually use to decide whether a change to Docent is an improvement, and it’s the metric behind almost every “our model beats theirs” claim you’ll read. But a pairwise judge run naively is worse than no judge, because it produces confident numbers that measure seating position instead of quality – and you’d never know unless you swapped the order and watched verdicts flip, as one to two of four did here on answers that were genuinely equal. Running both orders and counting only consistent verdicts is a cheap, mandatory discipline that turns a corrupt win-rate into a trustworthy one, and knowing about verbosity bias and non-transitivity keeps you from being fooled by a judge that rewards padding or from over-trusting a fragile multi-system ranking. This is also the natural bridge to the next lesson: every bias here is something you can only quantify by comparing the judge’s verdicts to human ones, which is exactly what calibration does.

Continue Building Your Skills#

You can now run pairwise comparisons for real: ask a judge which of two Docent answers is better, aggregate the verdicts into a win-rate, and – crucially – defend that win-rate against position bias by judging every pair in both orders and counting only what agrees. You watched verdicts flip on equal-quality answers just by swapping the order – one to two of four every run, with one pair flipping every single time – and you saw verbosity bias hand a vague answer a win it didn’t earn. That’s the honest state of a raw LLM judge: powerful, but biased in ways that quietly corrupt its numbers. Every fix so far – swap the order, normalize length, allow ties – has been a heuristic. In Lesson 4 you’ll stop guessing and start measuring: calibrate the judge against a set of human labels, compute judge-human agreement, and put a number on how much you can trust each verdict – the final step that makes both rubric scoring and pairwise preference safe to build an evaluation on.