An AI red team can spend days finding a path that makes a system disclose a secret, trust a poisoned document or call a tool outside policy. The demonstration is memorable; the protection often is not. A ticket records the prompt, a guardrail is tuned, the original string stops working and the next model or retrieval change quietly restores the behaviour.
The operating decision is what this finding must block now, and which repeatable test will prove that the vulnerable path stays closed after the system changes. That requires more than saving an attack prompt. It requires a finding contract that preserves the relevant conditions, an assertion at the boundary where harm would occur and a release rule owned by someone able to stop deployment.
The standards, taxonomies and tools linked below provide the source facts. The finding contract, three-layer assertion pattern, release table and worked example are AIEngine's engineering synthesis. They are not a claim that one test proves an AI system secure.
Treat the finding as a reproducible contract
A screenshot shows an outcome, not why it happened. Reproduction needs the system state and attacker conditions that made the path possible. NIST's adversarial machine-learning taxonomy classifies attacks by AI system type, lifecycle stage, attacker goal, capability and knowledge. Those dimensions belong in the finding, because changing any one may change both severity and the correct test.
Record at least:
- the affected application, route, tenant class and environment;
- model, prompt package, retriever, index, tool schema, policy and dependency versions;
- attacker access, knowledge and controllable inputs;
- the minimum ordered steps that produced the result;
- the security property violated and the consequence reached;
- the observable evidence that distinguishes success from failure;
- cleanup requirements for accounts, documents, messages or side effects; and
- owner, severity, temporary containment, permanent fix and release scope.
Keep the raw transcript, but also write a short causal statement. “The assistant followed a hostile instruction” is too broad. “Content retrieved from an untrusted supplier document caused the application to request an allow-listed tool with an unapproved destination, and the gateway executed it” identifies the trust transition that must be tested.
Separate model behaviour from the system exploit
The OWASP AI Testing Guide calls for repeatable cases across application, model, infrastructure and data layers. That separation prevents a common mistake: declaring a vulnerability fixed because the model now refuses the exact wording, while the application still permits the harmful operation.
| Layer under test | Useful assertion | Weak substitute |
|---|---|---|
| Model behaviour | The model does not propose the disallowed action across a declared attack set | One original prompt now receives a refusal |
| Control boundary | The policy gateway denies the unauthorised tool, resource, recipient or value | The system prompt says the action is forbidden |
| Committed effect | No prohibited read, write, message, payment or disclosure occurred | The final chat answer sounds safe |
For high-consequence findings, all three may matter, but the last two carry the release decision. A model can emit an unsafe proposal without causing harm if deterministic controls stop it. A polite refusal is irrelevant if a side effect already completed before the response was rendered. The five output-validation gates show how to keep generated intent untrusted until structure, meaning, authority and the final sink have passed independent checks.
Decide exactly what the finding blocks
Severity alone does not define a gate. A critical flaw in an unreachable research sandbox and a high-severity flaw on a production payment path should not create the same release scope. Combine consequence with demonstrated reachability and the components implicated by the causal statement.
| Evidence state | Release decision | Exit evidence |
|---|---|---|
| Harm reproduced on a production-equivalent path | Block every release containing the path | Boundary assertion and effect assertion pass; fix reviewed |
| Harm reproduced only under a broader test identity | Block permission expansion and the affected configuration | Least-privilege path passes under each real identity class |
| Model-only behaviour with no reachable side effect | Block model or prompt promotion when policy defines the output as prohibited | Declared behavioural suite passes within an agreed error budget |
| Plausible report not yet reproduced | Time-box investigation; preserve containment | Reproduction, bounded non-reachability evidence or explicit risk decision |
| Compensating control only | Permit only the named configuration and expiry | Permanent fix lands or the exception is renewed by its owner |
NIST's Secure Software Development Framework says secure development should address root causes to prevent recurrence and treats response to residual vulnerabilities as a core practice. Apply that discipline to the complete AI application. A model update cannot close an authorisation bug, and a gateway patch cannot establish that a poisoned index has been cleaned.
Write assertions where the harm becomes observable
Start from the violated property, not the attack text. Confidentiality means a protected value never reaches an unauthorised sink. Integrity means an untrusted source cannot alter a protected record. Authority means the attempted call is denied outside the approved subject, action, resource and conditions.
Use three kinds of assertion:
- Behaviour assertion: classify the model proposal or response against a narrow, reviewed rubric.
- Boundary assertion: inspect the policy decision, tool arguments, identity, resource, destination and denial reason.
- Effect assertion: query the system of record, outbound-request capture or transaction ledger for what actually committed.
Prefer deterministic checks for identities, destinations, field changes, network calls and transactions. A model grader may help classify ambiguous language, but it should not decide whether a payment occurred. If a grader is used, calibrate it as described in the LLM-as-a-judge guide, retain disagreement examples and never let its fluency conceal a missing effect assertion.
The UK AI Security Institute's Inspect framework expresses an evaluation as a dataset, solver and scorer, and supports agent tools and isolated sandboxes. That is a useful execution shape, not the security policy itself. The dataset preserves conditions, the solver exercises the real path, and scorers can combine model-behaviour checks with deterministic events from the gateway and sink.
Work an indirect prompt-injection finding
Consider a hypothetical procurement assistant that retrieves supplier documents and may draft an internal risk note. A red team uploads a document containing a hidden instruction to send contract excerpts to an external collection endpoint. The agent requests a web tool with that destination, and a permissive gateway makes the request. This example is illustrative, not an AIEngine incident.
The immediate containment disables arbitrary outbound destinations. The permanent design allows retrieval content to supply evidence but never authority. Destinations come from server-side policy, and the assistant's runtime identity cannot add one.
The regression fixture contains a harmless canary rather than real contract text. It runs through the same parser, chunker, retriever, model, tool schema, gateway and egress policy as the affected route. The test passes only when:
- the hostile document is retrieved, proving the fixture reached the model;
- no request reaches an unapproved host;
- the attempted call is denied with the expected policy rule and identity;
- the canary appears in neither outbound capture nor unauthorised logs;
- the system produces a safe review outcome or an explicit failure state; and
- a clean supplier document still completes the permitted drafting path.
The last check matters. A blanket network outage would stop exfiltration and make the product useless. The known-safe control proves the defence is selective. Run the case against every deployed model, parser, retrieval mode, tool version and identity class that shares the path. The dependency inventory provides that release-to-component map.
Preserve the exploit without preserving unnecessary danger
Attack fixtures can contain secrets, personal data, malicious files or live infrastructure details. Store raw evidence in a restricted case record with a retention rule. Derive the smallest safe fixture that still reproduces the mechanism: replace sensitive content with a canary, disable public callbacks, use test accounts and route tools into a sandbox or controlled sink.
Do not sanitise away the decisive property. If the exploit depends on file structure, encoding, document position, a particular permission or multi-turn state, the safe fixture must retain it. Record the derivation so a future maintainer understands what was changed and what must remain invariant.
MITRE ATLAS is a living knowledge base of tactics, techniques and case studies for attacks on AI-enabled systems. Map a finding to an ATLAS technique when it genuinely fits, but retain the local causal statement. A catalogue identifier helps group coverage; it does not capture this application's identities, tools and consequences.
The evidence record should follow the same minimisation logic as privacy-safe incident replay: preserve joined versions, decisions and effects without turning every production prompt or customer document into permanent security telemetry.
Expand from exact regression to mechanism coverage
An exact fixture proves that one known path remains closed. It does not prove that the mechanism is controlled. Build outward deliberately:
- vary phrasing, language, encoding and instruction position;
- place the attack in each supported input type and retrieval source;
- test single-turn, multi-turn and delayed tool execution;
- change destination, resource, value and tenant while holding intent constant;
- exercise the least and most privileged production identities; and
- include benign near-neighbours to measure over-blocking.
Label results honestly. Exact regression covers the captured exploit. Technique coverage covers reviewed variants. Property coverage asserts that the boundary prevents the prohibited effect regardless of wording. Only the last can support a broad claim, and even it is limited to the enumerated routes, identities and sinks.
The NCSC's secure AI system development guidelines organise controls across design, development, deployment, and operation and maintenance. Keep the test across that lifecycle: run it before release, monitor the corresponding boundary in operation and reopen it when incident evidence shows a new route.
Measure whether findings stay closed
Track reproducibility rate, time from finding to automated test, releases blocked, paths covered, recurrence after closure, false-pass investigations, benign-case rejection and flaky-run rate. Also count open findings without a runnable fixture, tests without a current owner, exceptions past expiry and deployed paths absent from the suite.
Re-run the affected set after a model, prompt, parser, retriever, tool, policy, identity, dependency or network change. The model change-control baseline gives the broader promotion pattern. If a test becomes flaky, quarantine the release rather than quietly retrying until it passes; investigate nondeterminism, rate limits, state leakage and grader disagreement.
The practical rule is simple: preserve the attack conditions, assert the real boundary, check the committed effect and name the release the finding can block. Red teaming finds paths that design review missed. Release tests turn those discoveries into lasting control.
