AI Shadow Traffic: Compare Safely Before the Canary

A candidate AI release can pass an offline evaluation and still fail on the requests, dependencies and load shapes found in production. Shadow traffic closes part of that evidence gap by sending a copy of a real request to the candidate while the approved production path remains the only path whose response reaches the user.

That definition sounds safe, but response suppression is only one property. The candidate may still receive personal data, consume scarce capacity, write to a shared cache, trigger a tool or create records through a credential it should never have held. A mirror at the gateway does not make the workload harmless.

The reader decision is: which production requests may be copied to a candidate, what authority may the shadow retain, and what paired evidence is sufficient to move from observation to a canary? This field guide treats shadowing as a controlled evidence lane. The platform behaviours and governance principles come from the official sources linked below; the isolation contract, comparison record and release gate are AIEngine's operating synthesis.

Separate shadowing from traffic shifting

Shadow, canary and A/B testing answer different questions because the candidate has different authority in each mode.

Amazon SageMaker's shadow-test documentation describes a production variant that answers all incoming requests and a shadow variant that receives a configurable replicated percentage without returning responses. Azure Machine Learning similarly states that mirrored traffic is copied to a deployment while clients continue to receive the existing deployment's result (Azure safe rollout guidance).

Service-mesh mirroring supplies the transport, not the whole experiment. Istio sends the mirror out of the primary request path and discards its response; it also marks the mirrored host or authority with a shadow suffix (Istio mirroring). The Kubernetes Gateway API likewise requires a mirrored backend's response to be ignored (HTTP request mirroring). Neither specification says the shadow cannot make its own outbound call. That is an application and infrastructure control.

Write the mirror contract before copying data

Start with eligible request classes, not an arbitrary percentage. For each class, record the production purpose, candidate purpose, fields copied, transformations, region, retention, access group and deletion path. Exclude a request unless the candidate is allowed to process every field it will receive.

A practical mirror contract contains:

• an immutable production and candidate release identity, including model, prompt, retrieval, tool and policy versions;
• the eligible populations and explicit exclusions, such as deletion requests, sealed investigations or unsupported languages;
• a deterministic sampling rule and maximum rate rather than a mutable dashboard percentage with no history;
• the permitted data fields, context sources, residency, log form and retention deadline;
• a list of destinations the shadow may reach and a deny rule for everything else;
• paired quality, safety, latency, reliability and cost measures with slice-level gates; and
• an owner, automatic stop conditions, maximum duration and evidence required to progress.

Copying is still processing. The ICO's current guide to the UK GDPR principles restates purpose limitation, data minimisation, storage limitation, security and accountability. Legal applicability requires qualified review, but the engineering consequence is direct: a shadow run does not justify copying every payload or keeping two full response logs indefinitely. Redact or tokenise before the fork where possible, retain structured comparison features instead of raw content where they answer the test, and propagate erasure into both lanes.

Make the candidate unable to perform business actions

The strongest shadow is useful even if its output is hostile. Give it a separate workload identity, no production write credential and no route to mutable business systems. Replace email, payment, ticketing, storage and workflow tools with recorders that validate the proposed call and store a non-executable representation for comparison.

Do not rely on a prompt that says “do not act”, a request header or the fact that the gateway discards the final response. Enforce the boundary at every effect-capable service. The AI output validation guide explains why a typed proposal still needs current authorisation, state and consequence checks outside the model; in a shadow lane, those gates should end at validation rather than commit.

Network isolation adds a second line of defence. Kubernetes documents that pods are non-isolated for egress by default and become egress-isolated only when an implemented NetworkPolicy selects them with the relevant policy type (Kubernetes Network Policies). A default-deny policy is useful only when the cluster's network plugin actually enforces it. Pair network controls with separate service accounts, read-only data access, disabled background jobs, isolated caches and destination-side rejection of shadow identity.

Also test the boundary. Seed the evaluation with prompts and retrieved documents that attempt to send messages, mutate records or acquire credentials. Any attempted business write is a release failure even when the recorder catches it; a write that reaches a real destination is an incident.

Pair the two paths without leaking identity

Comparison requires a stable join between production and candidate execution. Create an opaque comparison identifier at the fork, then give each lane its own child execution identity. The W3C Trace Context recommendation supports cross-system correlation but explicitly says that trace fields must not contain personally identifiable or otherwise sensitive information. Do not place a customer ID, email address, account number or prompt fragment in a trace header.

For each pair, preserve:

Run both lanes independently from the same approved input snapshot. Do not feed the production answer into the candidate or let the candidate reuse a response cache populated by production; either shortcut destroys the paired test. Preserve enough privacy-safe execution detail to reconstruct anomalies using the approach in the AI incident replay guide.

Sample for the decision, not the average request

A random one-percent mirror can miss rare but consequential work. Define strata for important task, language, source, complexity and consequence classes, then use stable hashing to select requests within each eligible stratum. Stable selection makes the policy reproducible and prevents autoscaling or retries from silently changing the sample.

Cap mirrored work separately by requests, tokens, bytes, concurrency and destination budget. Azure currently limits mirrored traffic to one deployment and at most 50 percent for its managed online endpoints, partly to limit bandwidth-quota impact; that product limit is not a safe default for another architecture. Start from the capacity the primary path can lose without breaching its objective. Measure gateway latency and error rates for non-mirrored as well as mirrored requests so the experiment cannot hide harm to production.

Track pair completeness. A candidate that drops the hardest requests may look fast and accurate among completed pairs. Report selected, dispatched, started, completed, timed out, cancelled and unpaired counts by stratum. Treat missing candidate results as outcomes, not rows to remove.

Define comparison rules before seeing the outputs

Predeclare how each result will be judged. Deterministic controls should cover schema validity, citation existence, policy route, prohibited content and attempted tool calls. Human or model-based scoring can assess usefulness and evidence quality, but blind reviewers to variant where practical and calibrate any automated judge against expert-labelled disagreements. The LLM-as-a-judge calibration guide covers position bias, rubric design and disagreement review.

Use paired differences rather than two unrelated averages: did the candidate improve, regress or tie on the same request? Report uncertainty and slices, not only a win rate. The evaluation sample-size guide explains why zero observed severe failures in a small sample is not evidence of zero risk.

A useful gate separates three claims:

• non-inferior guardrails: no material regression in severe error, unsafe route, privacy control or contract validity;
• operational fitness: candidate latency, failure rate, saturation and cost remain inside declared limits without harming the primary path; and
• decision value: a specified quality measure improves enough to justify exposing a bounded canary.

Work through a support-drafting candidate

Consider a customer-support assistant that retrieves approved policy and drafts a reply for an internal reviewer. Production may return a draft to the reviewer but cannot send it to the customer. A candidate changes both the retrieval model and the response model.

The mirror contract excludes identity-verification cases and copies only the fields already approved for model processing. The fork assigns an opaque comparison ID and resolves one immutable policy snapshot for both lanes. The candidate runs with its own read-only index, isolated cache and a tool broker containing only recorders. Even if it proposes “send email”, no credential or network route can reach the messaging service.

The team samples ordinary, long, multilingual and policy-conflict cases. It compares citation validity, governing-policy selection, severe unsupported claims, reviewer rubric scores, time to first usable draft, total latency and cost on the same requests. Timeouts remain failures. Reviewers see shuffled outputs without variant labels, and difficult disagreements go to a policy owner.

Suppose the candidate improves paired usefulness but times out twice as often on long conversations and retrieves an expired policy in one small product line. The correct result is not an averaged “win”. Keep it in shadow, fix temporal filtering and the long-context path, then rerun the affected strata. The change belongs in the immutable release record described in the model change-control guide.

Move to a canary only for evidence shadowing can collect

Shadow traffic cannot show whether users trust the answer, whether reviewers become complacent, whether a new interface changes escalation, or whether business outcomes improve. The candidate was deliberately prevented from causing those effects. It can also overstate latency if it runs at lower priority—or understate it if it receives warm capacity that a real rollout will not have.

NIST's AI RMF Playbook recommends measuring performance in development and deployment contexts, documenting tested conditions and failure behaviour, monitoring in real time and comparing evidence with risk tolerances. Shadow evidence is one deployment context, not a complete safety case.

When the shadow gate passes, create a separate canary release with an exposure cap, eligible population, staffed fallback, live outcome measures and tested rollback. Amazon's deployment-guardrail documentation distinguishes canary traffic shifting, where a small portion is served by the new fleet for a monitored period, from shadowing. Promotion should therefore be a new authority decision, never a switch that quietly lets the shadow's dormant tools become live.

The final review should show sample coverage, pair completeness, severe regressions, rejected write attempts, primary-path overhead, candidate latency and cost, retention compliance, reviewer disagreement and unresolved slices. Stop automatically for a real side effect, unauthorised data, primary-service harm, missing release identity or a breached severe-error gate.

The release question is not “did the candidate win more comparisons?” It is: did the mirror expose representative current work without expanding authority, did the paired evidence pass the predeclared gate, and which remaining claims require a canary? A well-designed shadow lane makes a candidate observable while keeping it operationally unable to matter—until an accountable release decision says otherwise.