From Model Output to Safe Action: Five Validation Gates

A model response becomes dangerous at the moment ordinary software mistakes it for authority. A string that is harmless in a review screen can become a browser payload, database command, file path, email recipient or tool argument when another component interprets it. The model does not need to be malicious. It only needs to be wrong in a way the receiving system did not expect.

The practical decision is not whether a model is generally trustworthy. It is whether this output may cross this boundary for this action. That decision should be made by conventional software using evidence and policy the model cannot rewrite.

This field guide presents five independent validation gates. The linked standards and guidance provide the source facts; the five-gate architecture is AIEngine's operating synthesis, not a formal standard or a claim that one validator makes an AI system safe.

The model sits outside the trust boundary

OWASP defines improper output handling as insufficient validation, sanitisation and handling before model output reaches downstream components. It identifies consequences including cross-site scripting, server-side request forgery, privilege escalation and remote code execution. Its useful design instruction is blunt: treat the model like another untrusted user.

That remains true when the application generated the prompt and the provider returned valid JSON. An attacker can influence model output through a customer message, retrieved page, email, document or tool result. A benign output can also contain a fabricated identifier, stale fact, impossible total or action outside the requester's permissions.

The NCSC's secure-design guidance for AI systems says systems that let AI components trigger actions should restrict the possible actions, add non-AI fail-safes where necessary and apply least privilege. In other words, model behaviour and system authority are separate engineering concerns.

This is the same boundary used in a controlled agent decision-rights map: the model may interpret and propose, while identity, policy, validation and commitment remain enforceable outside probabilistic reasoning.

Use five gates, not one confidence score

A confidence value cannot prove that a response completed, obeyed a contract, matched source evidence, stayed within authority and reached its destination safely. Give each property its own gate and reject the output when any required property is absent.

The order matters for efficiency, not for importance. Cheap parsing should happen before expensive source checks. Authority should be rechecked immediately before commitment because permissions or limits can change while the model is working.

Gate 1: verify the response envelope

Start with facts about the call, not the prose inside it. Record the provider request ID, selected model or deployment, status, elapsed time, completion condition and whether the response was refused, filtered, truncated or interrupted. Do not parse a partial stream as though it were the final business object.

OpenAI's Structured Outputs guide documents refusals as a distinct response state that consuming code must handle. Applications also need explicit paths for empty content and other non-parseable outcomes. The general principle is provider-independent: an exceptional envelope must never fall through to the success parser.

Retry policy belongs here, but only for operations known to be safe to repeat. A timeout does not prove that a tool or downstream service made no change. Use the AI degradation ladder to choose between a bounded retry, reduced service, deterministic fallback, human queue and fail-closed response.

Gate 2: enforce a narrow structural contract

Valid JSON is only a transport format. The contract must also define the expected root type, every required field, permitted keys, enums, lengths, numeric ranges, array limits, formats and nullable states.

The official JSON Schema explanation of applicability shows why incomplete schemas can appear to pass: properties do not become mandatory unless they are listed as required, and unexpected keys remain possible unless additional properties are constrained. The schema must also declare the root type for object-only rules to apply as intended.

Provider-native constraints are valuable. OpenAI states that Structured Outputs adheres to a supplied supported schema, while ordinary JSON mode guarantees valid JSON but not schema adherence. Its current function-calling guidance recommends strict mode and requires all fields to be marked required plus additional properties set to false for each object.

That is a strong syntax gate, not a truth gate. A schema can require an invoice total to be a positive number; it cannot establish that the number matches the invoice, that the supplier exists or that payment is due. Keep the runtime validator even when the provider constrains generation, and version the schema alongside the prompt, model, tools and policy in model change control.

Gate 3: validate meaning against evidence

Semantic checks ask whether individually valid fields make sense together and agree with authoritative sources. MITRE's CWE-20 guidance on input validation includes consistency, derived types, ranges, authenticity and conformance to domain-specific business rules among the properties a system may need to verify. It recommends accepting known-good values rather than relying only on a list of bad patterns.

For model-backed workflows, that means the validator should:

• recalculate totals, dates, units and checksums rather than accepting the model's arithmetic;
• resolve customer, supplier, product and document IDs against the authorised system of record;
• verify that cited evidence exists, was actually retrieved and supports the proposed field;
• reject stale or superseded policy and source versions;
• compare related fields for contradictions and impossible combinations; and
• express an abstain or review state when evidence is missing.

NIST's AI RMF Core says AI output should be interpreted within its identified context and calls for documented test, evaluation, verification and validation processes. Our implementation inference is that generic model quality cannot substitute for task-level rules. The same answer may be acceptable as a search hint and unacceptable as a committed account change.

Retrieval-backed systems need both source validation and answer validation. The enterprise retrieval-quality audit separates permission, retrieval, evidence coverage and answer quality so a fluent response cannot hide a missing or unauthorised source.

Gate 4: authorise the action, not the conversation

A tool schema describes what arguments look like. It does not prove that the current user may invoke the tool, see the referenced record or choose the requested value. Authorisation must use authenticated identities and current source-system policy at every consequential call.

Build a small allow-list of capabilities such as “create a draft case note for this case” rather than giving a model generic database or filesystem access. Bind the workload identity to the initiating user and tenant. Check record scope, field-level permission, transaction limits, recipient domains, approval state and separation of duties. Credentials should be held by the tool service, not placed in prompts or retrievable content.

Treat prompt injection as an attempt to produce an unauthorised proposal, not as a magical way to grant permission. An email can tell the model to export a client list, but the export service should still deny a workload with no export capability. A second model saying “safe” is not an authorisation service because it remains probabilistic and can be influenced by the same context.

Gate 5: protect the sink and prove the commit

Even a true, permitted value can be unsafe when inserted into the wrong interpreter. Render model text as text by default. If rich HTML is required, sanitise it with a maintained allow-list. Validate URLs and schemes. Use prepared statements for database values, fixed command interfaces instead of shell construction, bounded storage keys instead of generated paths and sandboxing for untrusted code.

OWASP's cross-site scripting prevention guidance explains that HTML, attributes, URLs, CSS and JavaScript require different contextual handling because browsers parse them differently. Its SQL injection prevention guidance prefers parameterised queries over string concatenation. These are ordinary application-security controls; AI output does not create an exception.

For side effects, send a typed command with an idempotency key and expected prior state. Require the destination to return a committed identifier and resulting version. Reconcile that response with the intended record before reporting success. If the result is unknown, preserve the correlation ID and investigate; do not ask the model to infer from a timeout that nothing happened.

Worked example: invoice intake without payment authority

Consider an assistant that reads an invoice and proposes a draft voucher. The response gate confirms one complete extraction. The structure gate permits only supplier ID, invoice reference, date, currency, net, tax, gross, purchase-order reference, evidence spans and a review reason.

The meaning gate then resolves the supplier and purchase order, checks currency, recomputes net plus tax, detects duplicate references and confirms every extracted amount against a cited page region. A new bank account in the document is recorded as a discrepancy, never as master-data truth.

The authority gate confirms that the user can submit a draft for this entity but cannot change supplier details or release money. The sink gate writes parameterised fields to the voucher-draft service with an idempotency key, receives the draft ID and compares the stored result with the proposal.

The model never obtains payment credentials. A perfect extraction still cannot post or pay because those capabilities were not granted. An invalid total goes to a visible exception queue with the source page and failed rule, rather than being silently corrected or repeatedly regenerated.

Make rejection a designed product state

Validation that can only return “error” will be bypassed under operational pressure. Return a typed reason such as incomplete response, schema violation, missing evidence, policy denial, unsafe destination or uncertain commit. Show operators what failed, retain the minimum evidence needed to investigate and provide an approved next step.

Do not automatically feed rejected output back to the model until it passes. Repeated generation may change the surface value without repairing absent evidence or authority. Some failures can be narrowed safely; others require a person or a stop.

Keep separate counts for accidental quality failures and suspected adversarial activity. Avoid logging raw prompts, personal data or secrets by default. The AI assurance evidence-pack guide provides a wider evidence register for claims, tests, versions, exceptions and accountable decisions.

Test the crossings and measure escapes

Build tests around every place output changes interpretation: JSON parser, browser, email renderer, URL fetcher, database, file store and tool service. Include missing and extra fields, type coercion, extreme lengths, Unicode edge cases, duplicate keys, hostile Markdown or HTML, crafted URLs, stale records, cross-tenant IDs, revoked permissions, prompt injection in retrieved content, repeated requests and timeouts after an apparent commit.

Measure properties of the control system, not how reassuring the model sounds:

• response-envelope failure and truncation rate;
• structural rejection rate by schema version and field;
• semantic rejection rate by rule and evidence source;
• unauthorised proposal and blocked-action rate;
• false acceptance found by human sampling or reconciliation;
• unknown, duplicate and compensated commit rate;
• reviewer queue age and time to correct a rejected case; and
• validation latency and cost per completed, compliant outcome.

Re-run boundary tests after any model, prompt, schema, retrieval, tool, identity, policy or destination change. Pause automated commitment if a consequential escape reaches a sink, authorisation cannot be checked, evidence provenance is lost, rejection queues exceed review capacity or reconciliation cannot distinguish success from failure.

Decision

Let a model output trigger action only when the application can prove five separate facts: a complete response arrived, the value matches a narrow contract, its meaning is supported by current evidence, the authenticated identities have authority and the destination can consume and commit it safely.

Structured generation reduces malformed output. It does not confer truth, permission or safe interpretation. Those properties belong to deterministic validators, source systems and accountable owners—and every failed gate needs a visible route that preserves control.

Primary source trail

• OWASP: LLM05:2025 Improper Output Handling
• OpenAI: Structured model outputs
• OpenAI: Function calling and strict mode
• JSON Schema: Applicability fundamentals
• MITRE: CWE-20 Improper Input Validation
• NCSC: Secure design for AI systems
• NIST: AI Risk Management Framework Core
• OWASP: Cross Site Scripting Prevention Cheat Sheet
• OWASP: SQL Injection Prevention Cheat Sheet