LLM-as-a-Judge: Calibrate the Grader Before the Score

An automated evaluator can turn a slow model review into thousands of scores before lunch. It can also turn an unclear rubric, a position preference or a factual blind spot into a confident release decision at the same speed. The danger is not simply that an LLM judge may be wrong. It is that the score looks objective after the assumptions that produced it have disappeared.

The operating decision is which judgements may be automated, what authority the judge receives and what evidence must exist before its score can block or advance a release. A useful judge is a calibrated measurement component inside an evaluation system. It is not a substitute for executable checks, domain expertise or accountable approval.

The studies and guidance linked below supply the source facts. The authority ladder, calibration packet, worked example and release measures are AIEngine's engineering synthesis. They are not a claim that one judge model, prompt or agreement percentage is safe for every task.

Decide what the judge is allowed to decide

“Use an LLM to grade the answers” hides several different jobs. Give each judge instance one declared authority:

• Annotate: produce a criterion result and evidence for later analysis.
• Prioritise: select likely failures for human review without disposing of the rest.
• Compare: rank two candidate releases on a bounded criterion.
• Gate: pass, fail or abstain against a tested release rule.
• Adjudicate: make the final consequential decision.

The first three can often be automated after validation. A release gate needs stronger calibration and a safe abstention path. Final adjudication should stay with a person or deterministic authority whenever the decision affects rights, safety, money, access or an irreversible action. The UK automated-decision guide covers the wider legal and redress questions; this article focuses on engineering the measurement itself.

NIST's AI RMF Core calls for objective, repeatable or scalable testing and for documenting test sets, metrics and tools. It also recommends independent assessors and domain input where appropriate. That is a useful boundary: automation may scale a measurement, but the organisation still has to show that the measurement means what its release decision assumes.

Use the cheapest valid measurement first

An LLM judge should not grade a property that code can determine more reliably. Split the evaluation before writing a judge prompt.

Anthropic's current evaluation guidance similarly places code-based grading ahead of human and LLM grading when the task permits it, and says an LLM grader should be tested for reliability before it is scaled. The practical lesson is not that human grading is always better. It is that every criterion should use the most direct valid measurement.

This separation also stops a polished answer from compensating for a hard failure. If an assistant invents a source, a high tone score must not rescue it. Apply the same principle used in the five output-validation gates: mandatory boundaries pass independently before preference scores matter.

Turn product intent into observable rubrics

“Helpful”, “accurate” and “professional” are product aspirations, not yet grading instructions. A rubric becomes testable when it names the evidence, observable condition, allowed result and consequence.

For each criterion, record:

• the user or business outcome it protects;
• the material the judge may inspect;
• pass, fail and uncertain definitions;
• hard disqualifiers that cannot be averaged away;
• examples near the boundary, including legitimate ties;
• slices where the rubric may behave differently, such as language, document type or user group; and
• the action attached to each result.

Prefer several narrow decisions over one impressionistic score. A one-to-five “quality” grade can conceal a factual failure behind good style and gives neighbouring numbers no stable meaning. Binary or ordinal results are not automatically valid, but their disagreements are easier to diagnose when the rubric states exactly what changes the label.

The 2023 G-Eval paper reported improved human correlation for its structured, criterion-driven approach on summarisation and dialogue tasks, while also warning about bias towards LLM-generated text. That is evidence for explicit criteria in those experiments, not permission to copy one prompt into a different domain.

Blind the comparison and attack the judge

The foundational MT-Bench and Chatbot Arena study tested pairwise, single-answer and reference-guided judging. In its settings, a strong judge often agreed with human preferences, but the paper also documented position, verbosity and self-enhancement biases plus failures on reasoning questions. A later systematic position-bias study found that position effects vary by judge, task and the quality gap between candidates.

Treat those as test categories, not folklore. Before a judge can gate a release:

• hide model and supplier identities from the grading prompt;
• evaluate pairwise candidates in both orders and mark a flip as disagreement, not a tie manufactured after the fact;
• repeat a sample to measure run-to-run stability;
• add a longer answer that repeats itself without adding value;
• add a fluent answer containing one known factual or logical defect;
• include outputs produced by the same model family as the judge and by alternatives;
• vary harmless style, formatting and confidence language while preserving meaning; and
• test prompt injection inside candidate responses and referenced documents.

The judge should receive candidate content as untrusted evaluation data, clearly separated from its grading instructions. Its explanation may help debugging, but an eloquent rationale does not prove the label. Store the criterion result, cited evidence, grader version and test identity so reviewers can inspect the decision without treating hidden chain-of-thought as an audit record.

Calibrate against people and known answers

Agreement is meaningful only against a defensible reference. Build a calibration set that combines:

• domain-expert labels for genuinely judgement-dependent cases;
• executable or known-answer cases for facts, logic, calculations and code;
• boundary cases on which qualified reviewers may reasonably disagree;
• sentinel pairs with one deliberate failure; and
• representative production slices, not just easy examples written by the development team.

Keep expert labels blind to candidate identity where possible. Allow reviewers to mark tie, insufficient evidence and rubric defect. Resolve a sample of disagreements through documented adjudication, then repair unclear rubrics rather than forcing every dispute into a winner.

OpenAI's GDPval methodology provides a useful source example: occupational experts blindly compared deliverables against detailed rubrics, while the automated grader was described as experimental and not reliable enough to replace those experts. The transferable principle is the separation between a scalable prediction of expert preference and the expert standard used to validate it.

Calibration must be reported as a confusion table by criterion and slice, not one blended agreement number. A judge that matches reviewers on tone but falsely accepts unsupported claims is not “mostly accurate” for a source-grounded assistant. Track false accepts, false rejects and abstentions against the consequence each creates.

Worked example: grade a source-based support answer

Consider a hypothetical support assistant answering a customer question from authorised product documentation. The release team is comparing a current and candidate system. This is a design example, not a claim about an AIEngine deployment.

This design refuses to let clarity outvote an access or evidence failure. It also gives the judge the minimum context needed for each criterion instead of asking one long prompt to assess security, truth, style and usefulness at once.

The enterprise RAG audit supplies the upstream corpus, permission and retrieval tests. The judge begins only after those layers establish which evidence was actually available.

Test the judge, not only the system under test

JudgeBench was created because matching preference labels does not establish factual or logical evaluation ability. Its ICLR 2025 paper uses challenging response pairs with objectively grounded preferences across knowledge, reasoning, mathematics and code, and reports that many strong judges struggled on that benchmark.

An internal judge therefore needs its own acceptance case. At minimum, measure:

• false-accept rate for each hard-fail criterion;
• false-reject rate and unnecessary review burden;
• order-flip rate for pairwise comparisons;
• repeat consistency under the declared sampling settings;
• agreement and error type by language, domain, length, risk and difficulty slice;
• abstention precision: whether uncertain cases really are harder or under-specified;
• human override and rubric-defect rates; and
• cost and latency per usable judgement.

Choose limits from error consequences, not a borrowed industry threshold. A content-ranking experiment may tolerate more disagreement than a release gate for regulated advice. When the calibration set is too small to estimate the material failure confidently, the honest result is “insufficient evidence”, not a wider automation scope.

Version the whole judging package

A judge result belongs to a package: judge model and endpoint, system instructions, rubric, examples, candidate presentation, reference material, tool code, parser, sampling settings and calibration-set revision. Change any one of them and the old acceptance evidence may no longer apply.

Record the package with the release using the model change-control baseline. Preserve the calibration report, known limitations, authorised slices and accountable owner in the AI assurance evidence pack. Do not silently follow a floating model alias for a release gate; if the served judge changes, re-run the sentinels and required calibration before accepting its verdicts.

Run the judge in shadow mode first. Compare its labels with the existing review path without changing outcomes. Expand authority criterion by criterion only when evidence supports it, and keep a quick route to disable automated gating without losing the underlying evaluation cases.

Revisit the instrument when its error pattern moves

Monitor the judge after release because candidate systems learn to produce different styles and production traffic moves beyond the calibration set. Review false accepts found by sampling, human overrides, order flips, slice-level drift, abstention volume and the share of failures caused by unclear rubrics.

Recalibrate when the judge model, prompt, rubric, evidence format, product task, candidate distribution or consequence changes. Pause automated gating when sentinels fail, order sensitivity rises, a material slice lacks coverage, reviewers overturn hard-fail decisions or the evaluation package cannot be reconstructed.

The score is the end of the measurement, not its evidence. Keep deterministic checks deterministic, make rubrics observable, blind comparisons, preserve legitimate uncertainty and validate the judge against people and known answers. Only then decide whether it may annotate, prioritise, compare or gate—and keep final authority outside the instrument where the consequence demands it.