An evaluation can improve every week while becoming less able to tell you whether the system improved. Engineers inspect failures, edit the prompt, replace retrieval documents, adjust a judge rubric and run the same cases again. The score rises. The test has now helped build the candidate it is supposed to assess.
The release decision is not simply whether the score passes. It is whether the evidence is independent enough to authorise this release, and what narrower claim remains when it is not. A familiar case can still be valuable as a regression test. It should not quietly retain the authority of an unseen acceptance test.
The standards, official guidance and research linked below provide the source facts. The four-corpus model, exposure record, evidence classes and worked example are AIEngine's engineering synthesis. They are not a claim that secrecy alone makes an evaluation representative, correct or safe.
Leakage is broader than copying the answer
Classical data leakage occurs when information unavailable at prediction time influences model construction. The current scikit-learn guidance says test data should not be used to make model choices and shows how even preprocessing learned from the complete dataset can produce optimistic performance. Generative systems widen that path: a case may influence the system prompt, retrieval corpus, few-shot examples, tool policy, output schema, grader rubric or human reviewer instructions without ever entering model training.
Foundation-model history adds another uncertainty. A public benchmark may already exist in pre-training data. Sainz and colleagues' contamination paper distinguishes exposure to annotation guidelines, raw text and labels, and notes that contamination can happen during pre-training, later training and iterative product improvement. Absence of verbatim recall is not proof that a model never saw the material.
Controlled 2026 experiments by Schaeffer and colleagues found that generative benchmark performance increased with deliberate test-set contamination, with effects depending on model size, dosage, further training and decoding. That study used MATH and research-scale models; it does not quantify the inflation in a particular production assistant. It does establish why a clean-looking score cannot repair an unknown exposure history.
Treat leakage as an evidence problem with several mechanisms:
| Exposure | Example | What becomes unreliable |
|---|---|---|
| Direct item exposure | A test prompt or reference answer is used for tuning | Performance on that item as evidence of generalisation |
| Semantic exposure | A paraphrase or near-duplicate case appears in development data | Independence despite no exact string match |
| Shared-source exposure | Train and test cases come from the same customer, incident, document or conversation | The apparent ability to handle a new entity or event |
| Procedure exposure | Developers learn the blind set's recurring traps or scoring rules | The test as an independent release challenge |
| Evaluator exposure | Judge prompts or thresholds are tuned on acceptance outputs | The reported score, even if the candidate never saw the cases |
Give each corpus one job
One growing eval file is convenient and ambiguous. Use separate corpora with different permissions and claims.
| Corpus | Who may see it | Permitted use | Evidence it can provide |
|---|---|---|---|
| Development | Builders and reviewers | Prompt, retrieval, tool and policy iteration | Diagnostic progress; no independent release claim |
| Grader calibration | Evaluation owners and labelled reviewers | Rubric, threshold and judge validation | Evidence that the measuring instrument behaves acceptably |
| Blind acceptance | A separate custodian or controlled runner | One run against a frozen candidate | Release, rollback or investigation evidence within the tested scope |
| Regression and incidents | Builders after a failure is disclosed | Prevent recurrence of known mechanisms | Evidence that a known path stays closed, not unseen-task performance |
This separation resolves a common conflict. OpenAI's evaluation best-practices guide recommends eval-driven development, held-out cases and continuous evaluation on every change. Those practices serve different purposes. A large visible regression suite supplies fast feedback; a smaller protected acceptance set preserves a decision gate. New production failures can enter regression immediately, while fresh or sequestered cases replenish acceptance.
The model change-control guide explains what belongs in the frozen behaviour package. The corpus contract adds who could see each example, when it was exposed and which decision the resulting score may support.
Map exposure across the whole system
The candidate is more than the endpoint name. Record the model snapshot, system and developer prompts, few-shot examples, retrieval snapshot, tool definitions, policy code, output validators, grader, rubric, threshold and reviewer instructions. If any of those components was changed after its owner examined an acceptance failure, the next run is not blind for that case.
Create an exposure record per corpus version:
- source, collection window, licence, consent and data class;
- unit of independence, such as customer, document family, incident, site or time period;
- content and semantic-duplicate checks against development and calibration sets;
- people, services, model providers and logging systems that could access inputs or labels;
- candidate, grader and rubric versions used for each run;
- disclosure date and destination when an item leaves the blind store; and
- resulting evidence class: blind acceptance, calibration, regression or invalid.
This is a practical extension of the NIST AI RMF Core. It calls for documented scientific-integrity and experimental-design considerations, documented test sets and tools, conditions similar to deployment, regular evaluation and involvement from people beyond front-line developers in assessment. See Map 2.3 and Measure 1.3–2.5 in the AI RMF Core.
Do not copy sensitive production conversations into a test vault by default. Minimise and de-identify cases where possible, preserve permitted provenance, control retention and use the joined identifiers in privacy-safe incident replay to connect a case to its evidence without turning the eval store into a second unrestricted customer database.
Split on the unit that can repeat
Random rows are often the wrong unit. Ten messages from one support conversation are not ten independent customer situations. Crops from one document page, translations of one question, records from one machine and multiple notes about one incident can leak their shared structure across a random split.
Choose the split boundary from the deployment claim:
- hold out complete customers or matters when the claim concerns new customers or cases;
- hold out document families and their derivatives when evaluating retrieval or extraction;
- split by later time when the system must handle future policy, language or behaviour;
- hold out sites, devices or suppliers when deployment extends to new operating domains; and
- preserve rare, harmful and abstention cases across the acceptance mix rather than allowing frequency alone to erase them.
Exact hashes catch copies, not meaning. Add canonical source identifiers, lineage and embedding-assisted near-duplicate review, then manually inspect uncertain clusters. Treat similarity as a triage signal rather than proof: two short policy questions can be semantically close while testing different rules, and a superficially different answer can expose the same underlying case.
Freeze first, then open the gate
Write the acceptance contract before the run: intended decision, population, case mix, metrics, uncertainty, subgroup or risk slices, critical-failure rules and stop conditions. Freeze the complete candidate and the measuring package. Only then allow the controlled runner to read blind inputs.
NIST's current AI Technology Evaluation programme uses blind data in a sequestered testbed so models can be compared on common tasks without the evaluation data becoming training material. A product team rarely needs a national laboratory, but it can reuse the control logic:
- a custodian stores inputs and reference material outside normal development access;
- the runner accepts only an immutable candidate identifier;
- prompts, outputs, tool traces and grader results are bound to one run identifier;
- aggregate results and approved failure details are released according to the contract; and
- any disclosed item is reclassified and, where needed, replaced before the next blind gate.
Sequestration protects independence, not validity. The blind cases must still reproduce the deployed task, including ordinary traffic, difficult edges, adversarial instructions and costly failure modes. Anthropic's evaluation guidance likewise recommends specific, measurable, task-relevant criteria, real-world task distributions, edge cases and validation of LLM-based grading before scale.
Decide what a leaked score may still do
Do not turn exposure into a binary ritual. Downgrade the claim explicitly.
| Evidence state | Permitted decision |
|---|---|
| Candidate and grader were frozen; task units are independent; set stayed blind | May gate the release within the documented population and uncertainty |
| Public benchmark may have entered foundation-model pre-training | Use for context; prefer fresh private tasks before a consequential go/no-go decision |
| Developers tuned after seeing failures | Reclassify revealed items as regression; run a replenished blind set for acceptance |
| Judge or threshold was tuned on acceptance outputs | Recalibrate on a separate labelled set, then score an untouched acceptance set |
| Cross-split duplicate or shared-source leakage is found | Invalidate the affected comparison and rebuild the split at the correct unit |
| Incident case is fully known | Use it to block recurrence of that mechanism, not to claim broad generalisation |
This also clarifies the relationship with red-team findings as release tests. A disclosed exploit should become a permanent regression. A protected family of related mechanisms can remain blind and test whether the system learned the boundary rather than the exact string.
Work a support-RAG release
Consider a hypothetical assistant that answers support questions from approved product and policy documents. Its builders use a development corpus to tune retrieval, citations, refusal language and escalation. A separate labelled set calibrates deterministic checks and the LLM judge; the judge never sees the blind answers during calibration.
The acceptance custodian groups historical material by conversation and source-document family, then holds out complete groups from a later time window. The mix includes ordinary questions, conflicting documents, superseded policy, missing evidence, prompt injection inside attachments and cases that require human escalation. The runner scores retrieval recall, citation support, unsupported claims, escalation recall, latency and cost. A critical unsupported policy answer fails the gate even if the average improves.
Suppose the candidate fails because it cites a superseded returns policy. The team receives the source family and trace, corrects version filtering and adds the revealed case to regression. It does not repeatedly call the same item “blind”. The custodian supplies a fresh policy-conflict case for the next acceptance run. The retrieval-quality audit supplies the component metrics; the blind gate protects the release claim.
Measure trust in the test, not only the model
Track acceptance coverage by task, risk, language, customer or operational slice; age of the protected set; direct and semantic overlap with development data; percentage of cases disclosed per release; time to replenish them; grader agreement with humans; score variance across repeated runs; and failures whose source unit crossed the split.
Also count process violations: unauthorised access to blind inputs, acceptance examples copied into tickets or logs, candidate changes after the freeze, and results published without their corpus and grader versions. A perfect model score beside an unversioned or exhausted holdout is weak evidence.
The rule is simple enough to audit: development tests help make the system better; blind acceptance tests decide whether the frozen system is ready. When a case crosses that boundary, keep its learning value, change its evidence label and replace the independent challenge before asking the score to authorise another release.
