AI Model Change Control: Upgrade Without Losing the Baseline

An AI model upgrade is rarely just a model upgrade. A new snapshot can change instruction following, refusal behaviour, tool selection, latency, token use and the shape of otherwise valid output. At the same time, a team may alter its system prompt, retrieval index, tool schema, safety policy or review interface to make the new model work. If the release succeeds, nobody can say which change helped. If it fails, there is no clean route back.

That is a change-control problem, not a leaderboard problem.

For a production AI service, the useful question is not “Is the new model better?” It is: “For this workflow, population and risk envelope, does this exact release produce more accepted outcomes at an acceptable total cost, without weakening a control we rely on?” The answer requires a retained baseline, a fixed evaluation corpus, layered measures, a limited release and a rollback that restores the whole service—not merely an old model identifier.

This guide provides that operating method for UK teams buying or building generative AI. It is not a claim that every change needs a committee or that one framework proves regulatory compliance. The depth of control should follow the consequence of failure.

Treat the release as a complete behaviour package

The model name is only one component in the behaviour users experience. Record the release as a package whose parts can be reconstructed together.

Give this package one internal release identifier. Link it to the source revision, configuration hashes, provider deployment, evaluation result and approver. A production trace should identify that release, not only a floating vendor alias.

The UK National Cyber Security Centre’s secure operation and maintenance guidance is explicit that changes to data, models or prompts can change system behaviour. It recommends update processes with testing and evaluation, preview access or versioned APIs, plus monitoring for sudden and gradual changes. That is a useful minimum even when the organisation is not training a model itself.

Provider lifecycle is an operational dependency

Pinned versions buy control; they do not buy permanence.

OpenAI’s API compatibility guidance says prompting behaviour can change between model snapshots and recommends pinned versions plus application evals for consistency. Microsoft Foundry exposes per-deployment upgrade policies: opt out until retirement, move when a new default appears, or move when the current version expires. Its guidance notes that model and API versions are separate contracts and that behaviour may change after an upgrade.

Google’s model lifecycle documentation similarly publishes release, retirement and replacement information and tells teams to test mission-critical functionality during migration. The specific dates will change; the design lesson will not. A supplier’s retirement clock is part of the service’s recovery plan.

Maintain a small lifecycle register with:

• the exact production model and deployment policy;
• the owner of provider notices and service-health alerts;
• known retirement or forced-upgrade dates;
• the latest safe date to begin evaluation;
• an approved fallback model or non-AI process;
• data-region, retention and contract differences in the fallback; and
• the maximum overlap cost the migration budget permits.

Do not let a procurement inbox become the only control. Provider notices need to create owned work with enough lead time to collect evidence.

Write the migration contract before seeing results

Testing becomes easy to manipulate when success is defined after the new model has produced attractive examples. Before running it, write a short migration contract.

The contract should state:

• Scope. Which users, tasks, languages, channels and actions are included—and which remain prohibited.
• Baseline. The complete current release package and the period used for production comparison.
• Candidate. The exact candidate package, with unrelated changes excluded where practical.
• Measures. Quality, control, latency, reliability, human-effort and cost definitions.
• Tolerance. The maximum acceptable deterioration for each critical measure, not just a target average gain.
• Release path. Offline replay, paired shadow, limited traffic and wider release conditions.
• Stop conditions. Events that pause or reverse the release immediately.
• Decision rights. Who can approve, pause, roll back and accept a documented residual risk.

Use risk tiers. A spelling assistant may need a lightweight record and sampled review. An agent that changes customer records, issues financial communications or handles safety-relevant material needs stronger test coverage, independent control checks and narrower release authority. The UK Government’s AI Playbook recommends full-lifecycle management, robust testing and monitoring, meaningful human control, and clear authority over model changes. Private-sector teams can adapt those principles without copying public-sector governance wholesale.

Build a corpus that can prove regression

A good regression corpus is not a folder of favourite prompts. It is a versioned set of representative inputs with expected evidence and adjudication rules.

Start with four sources:

• ordinary production cases sampled across real volume;
• rare but consequential cases selected by subject experts;
• past incidents, complaints, overrides and near misses; and
• deliberate attacks, malformed inputs, missing data and unavailable dependencies.

Preserve the raw input only where lawful and necessary. Remove or synthesise personal and confidential data when the evaluation does not require it. Attach metadata that matters to interpretation: workflow, language, document type, risk class, source quality and expected route. Do not let a large easy segment drown out a small high-consequence one.

Each case needs more than a reference answer. Record the required source or facts, unacceptable claims, required escalation, permitted tool calls and severity of error. For open-ended tasks, several answers may be acceptable; the invariant may be that every material claim is supported and a particular action is never taken.

Our enterprise RAG evaluation guide separates retrieval from generation so a model change does not receive credit for a newly indexed document—or blame for a missing one. Apply the same diagnostic discipline across the full release package.

Evaluate layers, not one blended score

An average “win rate” cannot establish production readiness. Report measures by layer and by important segment.

NIST’s 2026 report on deployed AI monitoring describes six distinct categories: functionality, operations, human factors, security, compliance and large-scale impacts. The report also highlights practical barriers such as drift detection, fragmented logging and scaling human monitoring. A migration dashboard should not pretend those categories collapse into one number.

Declare critical gates separately. A candidate with a meaningful quality gain should still fail if it performs a prohibited tool action, loses required evidence, exposes sensitive data or makes rollback impossible. Conversely, a small aggregate quality decline may be acceptable if it comes from a low-consequence segment and the release materially reduces severe errors elsewhere—but that trade must be explicit.

NIST’s Generative AI Profile includes change-management controls, pre-deployment testing, incident disclosure and documentation among the actions organisations can use to manage generative-AI risk. Treat those as connected evidence: test results without a reconstructable release are weak, while a perfect release log without meaningful tests is bureaucracy.

Use paired shadow before customer canary

Offline replay finds known failures. It cannot reproduce live distribution, dependency latency or reviewer response.

In paired shadow, the existing release serves the user while the candidate receives the same eligible case and its output is quarantined for comparison. Shadow only where duplicating the data and processing is permitted. Keep candidate tools read-only or simulated; a shadow agent must not send a second email, update a second ledger or create a second customer commitment.

Compare the two releases on the same case identifier. Investigate disagreements rather than rewarding whichever answer sounds more polished. The candidate may be right because it found better evidence, wrong because it ignored a limitation, or merely different because the task admits several valid forms.

Move to a customer canary only after the critical shadow gates pass. Cap the population and the candidate’s authority independently. For example, allow 5% of eligible summarisation traffic while keeping all record-changing tools behind the existing approval. Increase exposure based on observed outcome and control measures, not elapsed time alone.

An AI agent control room should show the active release, candidate exposure, control state, severe exceptions and rollback status. That makes the canary an observable operating condition rather than a flag known only to engineers.

Price quality, overlap and exceptions together

A faster or cheaper model can raise total cost if reviewers correct more outputs, retries increase or a larger context is needed to recover quality. A more expensive model can lower cost per accepted outcome if it reduces exceptions enough.

Calculate at least:

Cost per accepted outcome = model and platform cost + retrieval and tool cost + human review and correction + allocated monitoring and incident cost, divided by accepted outcomes.

During migration, add the temporary cost of parallel deployments, duplicated inference, evaluation, engineering and supplier commitments. State when that overlap ends. A candidate that needs a long dual-running period may create a cash and capacity issue even if its steady-state unit price is lower.

The production AI cost-stack guide provides the wider measurement model. For change control, retain both the expected range and the observed canary result. Do not approve a release on token price and discover later that review minutes doubled.

Roll back the service, not just the endpoint

A rollback is credible only if it has been exercised against the exact release.

The package should restore:

• the previous pinned model and API contract;
• its compatible prompt, examples and output parser;
• the matching retrieval index or schema version;
• prior tool definitions, permissions and approval gates;
• queue routing, user messaging and manual fallback; and
• the evidence needed to correct actions already taken.

Set a recovery objective in operational terms: how quickly must the harmful action stop, and how quickly must an acceptable service return? Those may be different. Revoking a tool can stop damage in seconds while cases temporarily route to trained staff. Restoring the full prior stack may take longer.

Exercise rollback before increasing authority. Include an expired credential, unavailable old endpoint and incompatible stored conversation in the rehearsal. If the previous provider version is approaching retirement, it is not a durable fallback; use a supported alternative or a non-AI path.

Record one decision that another team can audit

The release decision should fit in a concise evidence pack:

• business reason and affected workflow;
• baseline and candidate release identifiers;
• corpus version, coverage and known gaps;
• segment-level measures with confidence or sample limitations;
• all critical-gate results and unresolved exceptions;
• observed cost per accepted outcome and migration overlap;
• privacy, security, commercial and operational changes;
• canary scope, monitoring owners and stop thresholds;
• completed rollback evidence; and
• approval, expiry date and next review trigger.

Our AI assurance evidence-pack guide explains how to connect a claim to retained artefacts without turning governance into a document dump. For a model migration, the core claim is narrow: this exact candidate met these defined gates for this bounded use on this evidence.

A worked migration for an invoice assistant

Consider a hypothetical assistant that extracts invoice fields, retrieves purchase-order evidence and proposes an exception route. It cannot post a journal or release payment.

The current release has an 88% accepted-without-correction rate, a 1.2% material field-error rate and a median review time of 54 seconds. The candidate promises lower inference cost and structured output.

A disciplined test would not ask reviewers which output they prefer in twenty hand-picked examples. It would replay a fixed corpus segmented by supplier, scan quality, currency, language, tax treatment and exception type. It would test missing purchase orders, duplicate invoices, prompt injection inside attachments, unavailable retrieval and malformed tool responses. Critical gates would prohibit invented bank details, unsupported matches and routing that bypasses approval.

In shadow, both releases would receive the same eligible invoices, but the candidate’s proposed tools would remain simulated. The team would measure accepted outcome, material error, evidence-link correctness, correction time, P95 latency and fully loaded cost. If the candidate improves aggregate acceptance but doubles missed duplicate detection, it fails. If it reduces cost and review time while all severe gates hold, it can enter a capped advisory canary.

Rollback would restore the prior model, prompt, parser and retrieval contract together, then reconcile every candidate-handled case against the source invoice and purchase order. That is a tested service transition—not a hopeful edit to a configuration variable.

Monitor the release after the celebration

The first production week is part of evaluation, not proof that evaluation has ended. Compare candidate results with the declared baseline and watch for distribution shift, delayed complaints and reviewer adaptation.

Track:

• eligible volume and exposure by segment;
• accepted outcomes, severe errors and unsupported claims;
• prohibited or unexpectedly frequent tool proposals;
• reviewer disagreement, correction time and escalation;
• latency, timeout, schema and dependency failures;
• cost per accepted outcome and budget variance; and
• incidents, user reports, rollback events and unresolved evidence gaps.

Define who reads each measure and what action it triggers. A red metric without an owner or threshold is decoration. Preserve enough event evidence to investigate, but do not collect sensitive prompts indefinitely merely because storage is cheap.

Close the migration only when the candidate has met its live gates for the agreed coverage, the old commitments are safely retired, the lifecycle register is updated and the next material-change triggers are known. Model, prompt, corpus, policy, tool and workflow changes should each be capable of reopening assessment.

The durable advantage is not choosing the newest model first. It is being able to change a probabilistic dependency without losing the organisation’s memory of what good looked like, what failure cost, and how control returns when the new release does not behave as expected.