AI Reproducibility: A Seed Is Not a Replay Contract

“We kept the seed” sounds like a complete answer until another engineer tries to reproduce the result. The model name matches, the prompt looks unchanged and the same seed is present. Yet the generated answer differs. Was the original evaluation unreliable, did the serving stack change, or is the test demanding an equality the application never needed?

The useful decision comes before the rerun: what must repeat, within which controlled conditions, and what evidence would demonstrate that it did? A debugging fixture, a customer-facing decision and a claim about average model quality need different answers.

This guide focuses on inference and evaluation, especially self-hosted systems where the execution environment can be inspected. The linked documentation establishes specific implementation limits. The replay contract and test method below are AIEngine's engineering synthesis, not a universal determinism guarantee or an AIEngine benchmark result. Living documentation was checked on 12 September 2026; match configuration advice to the version actually deployed.

Choose the equality your decision needs

Do not put a single reproducible checkbox on a release. Name the object being compared and the allowed difference. These are practical contract categories, not claims about standardised research terminology.

An exact-output contract needs a comparison boundary. Comparing token IDs differs from comparing decoded text; comparing a parsed object differs from comparing JSON bytes. If key order is irrelevant, define canonicalisation before hashing. Do not quietly remove refusal markers, units or evidence references as cosmetic noise.

For behavioural equivalence, specify the invariants directly. Two summaries can use different sentences while preserving the same qualifications. Two tool proposals with different destination identifiers are not equivalent merely because their descriptions sound alike. Agreement is also separate from correctness: a consistently wrong answer passes an equality test.

Understand what the seed actually controls

A seed initialises a pseudorandom generator. It does not describe every input to the computation that consumes its output. The generator implementation, state progression and sequence of draws matter too.

NumPy's random-stream compatibility policy makes its conditions explicit: the generator and bit generator, seed, call sequence and arguments, build, environment and machine all participate. It even cautions that drawing several values in one array is not guaranteed to match repeated scalar calls. Reorganising a pipeline can therefore change random consumption without changing the seed field.

For an inference harness, give separate random streams to separately controlled purposes: case sampling, request generation, model sampling and any random evaluator. Record how a case receives its seed. Prefer a stable case identifier over its current position in a list so inserting a new case does not silently reassign every subsequent seed.

When resuming work halfway through a run, distinguish the initial seed from the generator's current state. A restarted process that repeats earlier draws is not necessarily equivalent to one restored at the interruption point. Document the resume procedure instead of assuming a seed is a checkpoint.

Avoid using one favourable seed as evidence of quality. A fixed seed is useful for locating a difference; a declared set of repeated trials is useful for understanding whether that difference survives ordinary sampling variation. Those are separate experiments.

Deterministic sampling is not deterministic execution

Choosing the highest-scoring token removes a sampling choice, but does not establish that the computed scores will be identical. If two candidates are nearly tied, a small numerical difference can change which one is selected. After that first different token, subsequent generation has different context. This is a mechanism to investigate, not a claim that every mismatch has a floating-point cause.

The PyTorch reproducibility note, updated in May 2026, explicitly excludes guarantees across releases, commits and platforms, including CPU versus GPU execution with identical seeds. It distinguishes seeding from selecting deterministic algorithms. Its deterministic-algorithm setting can use a supported alternative or raise an error for a known nondeterministic operation without one; it does not make an arbitrary surrounding application deterministic.

TensorFlow's operation-determinism documentation, last updated in April 2024, explains how parallel execution can change floating-point addition order. It requires compatible hardware, software and input-pipeline conditions, and warns that deterministic operation can reduce performance. It also distinguishes output determinism from latency, memory consumption and throughput, which the setting does not make deterministic.

Use a constrained diagnostic configuration when exactness helps isolate a bug. Then test the actual production configuration separately. A slower deterministic harness is not evidence that the high-throughput serving path behaves identically, and its latency figures should not be presented as production capacity.

Put scheduling and external state inside the boundary

An online request shares an execution system with other requests. Testing it alone may not reproduce its original scheduling conditions.

The current vLLM reproducibility guide does not promise reproducibility by default. It describes deterministic scheduling or batch invariance for offline use, and batch invariance for online use, while retaining same-hardware and same-version conditions. These are scoped controls, not evidence that a seed alone stabilises a service.

vLLM's separate batch-invariance documentation, dated August 2026, describes outputs independent of batch size and request order. When checked, the feature was beta, required NVIDIA compute capability 8.0 or higher, listed tested models and warned of performance impact. Confirm support and limitations for the installed release before treating the feature as an operational promise.

The application boundary extends further. Freeze the resolved prompt, retrieved evidence and tool responses needed by the experiment. A search query rerun against today's index is a new input, even if its text matches yesterday's query. A live tool that reads a changed record can legitimately alter the answer.

For hosted APIs, record the request configuration and any returned revision or system identifiers the provider exposes. Mark undisclosed execution details as unknown. If the supplier contract does not establish exact regeneration, choose a behavioural or evaluation contract that can actually be tested. Do not convert missing information into a promise.

Store a run manifest that can reconstruct the test

A run manifest is an index to the experiment's dependencies and comparison rules. It should identify recoverable artefacts, not merely give each mutable object a reassuring name.

Retain, as applicable:

• The application commit and immutable environment or container digest.
• Model weights, adapters, tokenizer and chat-template revisions or digests.
• Runtime, driver, accelerator configuration, numerical precision and selected backend.
• Resolved instructions, decoding parameters, stop rules and output limits.
• Case-set revision, preprocessing rules and the exact input ordering.
• Generator identities, seeds, stream assignment and resume-state procedure.
• Retrieval snapshots and controlled references to tool-response fixtures.
• Serving mode, concurrency scenario and relevant scheduling settings.
• Comparator version, tolerance rules, reference labels and evaluator configuration.
• Original outputs, failure states and links to the comparison results.

The MLflow Tracking documentation provides one implementation foundation: runs can record parameters, code versions, metrics and output artefacts. Using an experiment tracker does not automatically capture the entire boundary. Audit what the integration actually saves, especially preprocessing and external evidence.

A digest can establish that a recovered artefact matches; it cannot recover an artefact that no longer exists. Test permissions, retention and restoration as part of the manifest. Avoid collecting secrets or unrestricted production transcripts simply to improve replay convenience. Use permitted fixtures and controlled evidence references, following the separate privacy-safe incident replay guide when historical content is needed.

This run-level manifest is more specific than a model-change baseline. The baseline identifies the approved service; the manifest identifies the exact experiment performed against it.

Diagnose a mismatch with a controlled replay drill

Consider a synthetic support-ticket classifier that proposes a queue and a short explanation. It cannot send a customer message or modify a ticket. The purpose of the drill is to determine whether reruns preserve the proposed queue and required escalation flags. No deployment results are assumed here.

Create permitted fixtures containing ticket text, resolved instructions and frozen reference material. Define queue and escalation equality as exact, while allowing explanation wording to vary if its required facts and qualifications remain intact. Store the original output separately from every rerun.

Begin with the smallest controlled comparison:

• Rebuild the recorded environment and verify artefact digests before inference.
• Repeat the same case in fresh processes with identical settings and fixtures.
• Compare preprocessed inputs and token IDs before comparing generated answers.
• Repeat with the case alone, then with declared neighbouring requests and orderings.
• Change one supported execution setting at a time and retain every result.
• Restore live retrieval or tools only in a separately labelled integration test.

Suppose the queue remains equal but the explanation differs. That is an exact-text mismatch, not automatically a behavioural failure. Inspect the explanation against the predeclared criteria. If a changed explanation removes a required warning, the behavioural contract fails even though the queue is unchanged.

If mismatches appear only with different neighbouring requests, scheduling or batch-sensitive computation becomes a candidate cause. If input tokens already differ, investigate preprocessing or prompt construction first. A diagnostic configuration that removes a mismatch narrows the investigation; it does not, by itself, prove a particular kernel was responsible.

Keep replay tools read-only or simulated. An experiment that reproduces the original write against production can duplicate harm rather than explain it. Where original evidence is unavailable, report that boundary as unreconstructable and create a new prospective fixture. Do not label an approximate reconstruction as the historical run.

Evaluate variation without choosing the winner afterwards

Once the harness can explain its controlled boundary, use repeated runs to assess the application's quality claim. Exact agreement on one seed says little about performance over other draws or other cases.

Predeclare the case set, trial count, seed assignment, comparator and stopping rule. Pair baseline and candidate on the same cases and trial identifiers. Matching seeds can aid organisation, but does not guarantee equivalent random draws across different model implementations.

Keep case variation separate from run variation. Repeating one ticket a hundred times does not provide the coverage of a hundred different tickets. Report both how many distinct cases were tested and how many repeats each received. The evaluation sample-size guide explains why uncertainty and coverage belong beside an aggregate pass rate.

Inspect unstable cases rather than discarding them. They may expose an ambiguous reference label, a decision near a threshold, a missing qualification or genuine model variability. If the evaluator itself uses a model, retain its configuration and test its repeatability separately; otherwise changing judgements can masquerade as changing candidate quality.

Never rerun only the failures until the dashboard turns green. Preserve failed, timed-out and refused trials under a declared counting policy. Report quality, reproducibility and performance as separate results. A model can be reliably repeatable yet consistently incorrect, or useful on average while too unstable for a particular consequential decision.

Measure the promise and its cost

Track reconstruction success: the share of sampled manifests that another engineer can restore without undocumented files or personal knowledge. Track exact mismatch rate only inside the declared exactness boundary, and behavioural disagreement by meaningful case type. Record unknown environment fields and unavailable evidence as gaps, not successful comparisons.

Measure the cost of stricter execution settings on the intended workload: complete-request latency, throughput, resource use and maintenance effort. Where exactness is unnecessary, a well-specified behavioural contract may offer a better tradeoff. Where a critical control must not vary, enforce that control outside model improvisation rather than treating a seed as a safety mechanism.

Reopen the evidence when the runtime, hardware, numerical backend, tokenizer, preprocessing, serving mode or comparator changes. The quantised-inference release guide is especially relevant when a numerical optimisation alters the execution path. Preserve the previous result as evidence for its original conditions, not as a guarantee for the new ones.

The final claim should be narrow enough to falsify: these artefacts, under these conditions, satisfied this comparison rule across these trials. A seed belongs in that statement. It cannot replace the rest of it.