AI Incident Replay: Trace Outcomes Without Logging Everything

An AI incident often begins with a simple question that the telemetry cannot answer: why did this outcome occur? The application may have retained a full prompt yet lost the model revision, retrieved-record versions, policy decision or committed side effect. Another team may keep every message indefinitely and create a larger privacy and security problem without making the event reproducible.

The operating decision is which evidence should exist by default, which content should require escalation and how long each item remains useful. The answer is not “log everything” or “log nothing”. It is a deliberately joined replay packet that can reconstruct the system's path while keeping content capture exceptional.

The standards and guidance linked below supply the source facts. The replay packet, capture states and measures are AIEngine's engineering synthesis, not a formal standard or a claim that any trace makes a probabilistic model exactly reproducible.

Replay is an evidence problem, not prompt history

Exact regeneration is usually the wrong promise. A hosted model can change behind an alias, a stochastic sampler can choose different tokens, a search index can be updated and a tool can return a new result. Even a complete transcript may not recover the state that shaped the original outcome.

Operational replay has a narrower and more useful objective: reconstruct the request path, identify the versions and evidence used, verify the enforced decisions, locate the side effect and test whether the controls would reach the same disposition. That supports containment and correction without pretending that identical prose proves identical system behaviour.

NIST's AI RMF Core calls for post-deployment monitoring, incident response and recovery plans and says measurement and monitoring should be documented, repeatable and traceable. The NCSC's secure-operation guidance for AI systems similarly recommends monitoring inputs in line with privacy and data-protection requirements, observing outputs and performance, and accounting for behavioural changes caused by updates to data, models and prompts.

Those requirements point to versioned evidence, not indiscriminate transcripts. The same version set should already be controlled in an AI model change baseline; incident replay gives that baseline an operational trace to explain.

Start with one trace spine

Every component needs a shared correlation value before its local events can become a sequence. The W3C Trace Context Recommendation defines the interoperable traceparent and tracestate headers for joining a request across system boundaries. It also warns that personally identifiable or otherwise sensitive data should not be placed in trace data and that systems should consider privacy at trust boundaries.

Use an opaque, randomly generated trace identifier as the spine. Join provider request or response IDs, internal workflow IDs and committed record IDs to it in protected event storage; do not encode a customer email, document number, tenant name or case description into the header. A trace identifier locates evidence. It should not be the evidence itself.

Record these four joins explicitly:

• the authenticated actor and workload as stable pseudonymous identifiers, with the tenant and purpose kept in controlled fields;
• the application trace and span IDs carried across queue, retrieval, model, validator, tool and destination boundaries;
• the provider's request or response identifier and the actual returned model name, not only the requested alias; and
• the destination's idempotency key, committed object identifier and resulting version, including an unknown state when the commit cannot be confirmed.

An interaction identifier is also part of OWASP's recommended event attributes in its Logging Cheat Sheet. The practical value is join completeness: a model call with no link to its retrieval span or final write is an isolated receipt, not a replayable outcome.

Build the replay packet in six layers

The packet should answer a bounded question at each layer. Prefer immutable identifiers, versions, hashes and decisions. Store content only when that evidence cannot answer the incident question.

OpenTelemetry's current generative-AI span convention provides useful field-level foundations, including operation name, provider, requested and returned model, response identifier, finish reasons and token usage. The convention is still marked Development, so pin the semantic-convention version and test upgrades rather than assuming field stability.

This packet complements the five output-validation gates. Validation determines whether one proposal may cross a boundary; replay preserves the evidence that shows which gates ran, on which package, and what the destination actually committed.

Decide content capture separately from tracing

The same OpenTelemetry convention treats system instructions, input messages, output messages and tool arguments or results as opt-in because they are likely to contain sensitive information and can be high volume. Its production guidance suggests storing content externally with separate access controls when sensitivity or volume makes ordinary telemetry unsuitable.

That distinction should become an explicit capture policy:

Move upward only for a stated purpose, named owner and expiry. Redact before writing to ordinary telemetry rather than treating later cleanup as the primary control. Keep the content vault, key permissions and access audit separate from the observability store so access to latency charts does not silently grant access to conversations.

The ICO's data-minimisation guidance says personal data should be adequate, relevant and limited to what is necessary for the purpose. Its storage-limitation guidance says organisations should justify retention periods, review stored data and erase or anonymise it when no longer needed. Both pages note that the guidance is under review following the Data (Use and Access) Act 2025, so teams should check current obligations with their privacy owners rather than copy a universal retention number.

Make the evidence trustworthy

A complete-looking trace is not useful if an operator can rewrite it, clocks disagree or events vanish under load. OWASP advises protecting logs from tampering and unauthorised access, recording and monitoring access, sanitising event data, and testing logging failures and resource exhaustion. It also lists secrets, access tokens, passwords and sensitive personal data among values that should generally not be recorded directly.

Give the replay path ordinary security controls:

• append events to storage where amendments create a new, attributable record instead of replacing history;
• synchronise clocks and retain both event time and ingestion time so queue delays do not distort order;
• sign or hash critical event batches and monitor gaps, rejected writes and impossible sequence changes;
• restrict fields by role, with more control around identity joins and sealed-content pointers;
• mask or tokenise sensitive identifiers consistently enough to support an authorised join;
• version the event schema and preserve parsers for records still inside retention; and
• make telemetry failure visible to the product, using the graceful-degradation ladder when a consequential action cannot produce required evidence.

Sampling needs a policy of its own. W3C's sampled flag communicates a tracing decision, but the Recommendation cautions that it is not a security mechanism or a guarantee that every downstream system recorded the trace. Retain required decision and effect events deterministically for consequential workflows; sample high-volume diagnostic spans only after the minimum replay packet is secure.

Walk an incident without changing its history

Run the replay as an evidence procedure, not a fresh chat with the model:

• Freeze the relevant trace, access records and content-vault pointers under an incident identifier without broadening access.
• Verify sequence integrity, clock assumptions, join completeness and the production package versions.
• Reconstruct the request, authorised retrieval results, validator decisions, approvals and destination response as they existed at the event time.
• Compare the effect with the initiating purpose and expected prior state; reconcile the committed record independently.
• Re-run deterministic parsers, policy checks, calculations and validators against preserved inputs and versions where safe.
• If a model must be called again, label it as a new evaluation with its own trace and package. Never insert the new output into the historical record as though it were original.
• Record the cause, control gap, correction, affected traces, owner and evidence-backed closure decision.
• Release sealed content and resume ordinary deletion when its approved purpose ends, unless a documented hold remains.

This procedure also creates the incident portion of an AI assurance evidence pack: the claim is not merely that monitoring exists, but that a reviewer can follow an outcome from request to evidence, decision and effect.

Avoid six false forms of observability

The full-transcript archive. It increases breach impact and review burden while still missing policy, retrieval and commit state. Add structural evidence before adding content.

The metrics-only dashboard. Aggregate latency, errors and tokens can show that something changed but cannot identify the evidence or decision behind one outcome.

The prompt hash alone. A digest can prove equality with a candidate prompt, but cannot reveal the candidate when no controlled source exists. Retain the template version and safe input references needed to reconstruct it.

The provider receipt. A model request ID does not explain which local permissions, validator results, tool call or database version surrounded the call.

The mutable incident note. A manually edited narrative is interpretation. Link each conclusion to immutable events and preserve corrections as new records.

The permanent exception. Temporary full-content capture often survives its incident. Enforce expiry technically, monitor deletion and require fresh authority for extension.

Test replay before the incident

Choose representative consequential workflows and inject failures at every join: provider timeout, missing retrieval version, changed access decision, validator crash, approval revocation, duplicate tool request, destination timeout after commit, telemetry write failure and sealed-vault denial. Include malicious log delimiters, extreme field lengths, cross-tenant identifiers and secrets in tool results to verify sanitisation and isolation.

For each test, an operator who did not build the feature should be able to identify the package, recover the permitted evidence set, explain every control decision, reconcile the side effect and find the retention rule without database archaeology. If that is impossible, the evidence contract is incomplete even when dashboards are green.

Measure replay readiness

Track whether the evidence system itself works:

• trace join rate across retrieval, model, validation, tool and destination spans;
• replay-packet completeness by workflow and package version;
• unknown model, policy, source or committed-record version rate;
• median and high-percentile time to reconstruct a consequential outcome;
• percentage of traces escalated to redacted or sealed content, with reason and owner;
• detected sensitive-data leakage into ordinary telemetry;
• telemetry rejection, drop and integrity-failure rate;
• expired content or identifiers not deleted within policy; and
• incident closures that required an unsupported inference because evidence was absent.

Revisit the contract whenever a model, prompt, retrieval index, tool, policy, destination, event schema or legal purpose changes. Pause consequential automation when required decision or effect evidence cannot be written, joins repeatedly fail, the actual package is unknown, content capture escapes its approved boundary or reconciliation cannot establish what changed.

Decision

Do not choose between observability and privacy as if one requires a permanent transcript. Keep a minimal trace spine, version the executable package, record evidence references and enforced decisions, prove the resulting effect, and escalate to content only for a bounded purpose with separate controls and deletion.

A useful replay can explain what the system knew, what it was allowed to do, what controls decided and what finally changed. Anything more should earn its place through necessity; anything less should be treated as an operational gap.