An AI feature can fail while every server remains online. The model may time out, hit a capacity limit, return a structurally valid but unusable answer, lose the evidence it needs, or propose an action outside its authority. A generic “try again” handler treats these different conditions as one technical fault. That is how a small dependency problem becomes duplicate work, a growing queue or a confidently wrong customer outcome.
Graceful degradation asks a more useful question: what is the safest valuable service this product can still provide under this specific failure? Sometimes the answer is a cheaper model or a shorter response. Sometimes it is a deterministic rule, a saved draft, a visible human queue or no action at all.
This guide turns that choice into a failure ladder. It is for teams designing an AI-assisted workflow, not only infrastructure engineers operating an API. Reliability here is a product policy: the business must decide which promises can be reduced, which decisions can wait and which actions must stop.
Degradation is not the same as retrying
A retry attempts the same operation again because the fault is expected to be temporary. Degradation changes the service being offered so that the system does less work, assumes less authority or makes a narrower promise.
Google's Site Reliability Engineering guidance defines graceful degradation as reducing the work needed to serve a request. Its examples trade some response quality for lower computational cost under overload. The same mechanism applies to AI, but model-backed products add a second dimension: a response can be cheap and available yet not trustworthy enough for its intended use.
Our operating interpretation is therefore broader than load shedding. An AI service should degrade when any dependency required for a safe outcome is unavailable: model capacity, approved context, a policy rule, a tool, an audit trail or competent review. The trigger is not “the model is down.” It is “the evidence or control behind this service promise is missing.”
That makes degradation part of the full production AI cost stack. Manual capacity, queue reconciliation, secondary providers and failure testing all cost money. Omitting them does not remove the cost; it makes the first outage discover it.
Classify the failure before choosing the response
Start with five failure classes. They require different signals and should not share one catch-all fallback.
| Failure class | Observable evidence | Why a blind retry is unsafe | Safer first response |
|---|---|---|---|
| Transport | Connection reset, timeout before a response, provider 5xx | The original request may still have completed, especially if it invoked a tool | Retry only an idempotent operation within a bounded time budget |
| Capacity | Rate limit, overload response, exhausted concurrency | Immediate retries add load and can prolong the incident | Honour provider timing, shed low-priority work and open a circuit when thresholds persist |
| Quality | Invalid schema, missing citation, failed evaluation rule, low task-specific score | Repetition can produce a different answer without making it more reliable | Reduce scope, request missing evidence or route to review |
| Control | Permission denied, policy block, unavailable approval service, action outside limit | A retry cannot create legitimate authority and may look like bypass behaviour | Preserve the request and fail closed before the action |
| Observability | Missing trace, unknown model version, lost tool result, stale source timestamp | The team cannot reconstruct what happened or distinguish success from failure | Stop consequential automation and move to a visible exception queue |
Provider status codes help with the first two classes, but they are not a complete business policy. OpenAI's current error-code documentation distinguishes rate limits and temporary server overload from billing, spend and usage-limit errors that require operator action. Anthropic similarly documents 429 rate limits, 5xx errors, timeouts and a 529 overload condition, while permission, request-format and billing errors have different meanings.
The important boundary is between *transient* and *action-required*, not between “error” and “success.” A 200 response with an invalid invoice total belongs to the quality class. A 403 belongs to control. Neither becomes safe because another sample might work.
Define the service ladder before the incident
A degradation ladder gives the runtime an approved lower rung instead of asking an on-call engineer to invent one under pressure.
| Rung | Service promise | Appropriate use | Prohibited shortcut |
|---|---|---|---|
| 1. Full service | Evaluated model, approved context and tools operate inside normal limits | All health, quality and control signals are present | Treating provider availability as proof of output quality |
| 2. Reduced AI | Smaller scope, shorter context, cheaper model or read-only tools | The reduced mode has its own evaluation and disclosure | Quietly switching to an unevaluated provider or region |
| 3. Deterministic fallback | Rules, templates, cached approved facts or ordinary software complete a narrower task | The rule can make the promise without model judgement | Presenting stale or partial content as current and complete |
| 4. Human queue | Input and evidence are preserved for authorised review | Delay is safer than an uncertain decision | Building an unbounded queue with no owner or response target |
| 5. Fail closed | No consequential action occurs; the user receives an honest status and recovery route | Authority, evidence, safety or reversibility is absent | Returning a plausible success message while work was dropped |
Each rung needs an explicit entry trigger, exit test, owner, maximum duration and user message. Reduced AI is not automatically safer: a smaller model can have different refusal, language or extraction behaviour. A secondary provider can move data to another jurisdiction or retain it differently. Evaluate each rung as a release package using the same discipline as AI model change control.
The ladder should also follow consequence. A marketing draft may move from full service to a template. A payment release should stop when the approval or audit service is unavailable. “Always available” is not a credible objective when the remaining path cannot lawfully or safely complete the action.
Bound retries with deadlines, jitter and idempotency
HTTP already provides a useful coordination signal. RFC 9110 defines Retry-After as either a date or a number of seconds to wait; a service can send it with 503 to suggest when a temporary overload may ease. OpenAI's rate-limit guidance says custom clients should honour a valid Retry-After value, otherwise use exponential backoff with jitter, while limiting both attempts and total retry time. It also warns that SDKs may already retry, so adding another loop can multiply attempts.
Set one end-to-end deadline from the user's perspective, then allocate smaller budgets to model, retrieval and tool calls. Three retries of a 20-second model call do not fit a 30-second interaction. When the deadline is spent, move down the ladder; do not keep invisible work alive after the user has reasonably acted again.
State-changing operations need another boundary: the system must know whether the first attempt took effect. Stripe's idempotent-request design illustrates the principle: a unique key lets the server recognise a repeated create or update request and return the original result rather than perform the operation twice. An AI workflow should generate the operation identity before calling the model or tool and preserve it through retries, queues and recovery.
Idempotency prevents duplicate execution; it does not reverse a valid but unwanted action. Before granting tool authority, label every step as read-only, idempotent, compensable or irreversible. That label should constrain which rung is available.
Use two circuit breakers, not one
A technical circuit breaker protects the application and its dependency from repeated calls during a persistent fault. Microsoft's Circuit Breaker pattern separates three states: closed calls the dependency, open fails immediately, and half-open permits limited probes to test recovery. This differs from retry: retry expects eventual success, while a breaker stops calls that are unlikely to work.
An AI product also needs a business breaker. It opens when the service is technically responsive but the output cannot support the intended promise. Triggers might include:
- source coverage below the approved minimum;
- a required policy or price list older than its validity window;
- schema-valid outputs failing a reconciliation rule;
- unusual growth in human overrides or downstream reversals;
- loss of the request, model, source or tool identifiers needed for audit; or
- reviewer queue age exceeding the point at which the decision remains useful.
Keep the breakers separate. The technical breaker may route a summarisation request to an evaluated secondary model. The business breaker may forbid that route for a regulated decision because the secondary mode lacks the required evidence. A single global “AI healthy” flag cannot express that difference.
Recovery should be cautious. Half-open traffic must be small enough not to recreate overload, and business recovery needs representative quality checks rather than a successful health endpoint. Record every state transition and manual override in the AI assurance evidence pack.
Treat partial actions as a reconciliation problem
Agentic workflows can complete step one and fail at step two. A model may prepare a supplier record, a tool may create it, and the response may disappear before the workflow stores confirmation. Retrying the whole sequence can create a duplicate; assuming failure can leave an unmanaged record.
Microsoft's Compensating Transaction pattern explains why distributed work cannot always be restored by copying old state over new concurrent changes. The workflow must record each completed step and its business-specific undo or correction action. Some steps are irreversible, and compensation can itself fail.
For every consequential tool call, retain:
- one operation and correlation identifier;
- the intended action, parameters, actor and authority;
- preconditions checked immediately before execution;
- the tool's response and external record identifier;
- whether the result is confirmed, unknown, failed or compensated;
- the permitted reconciliation or compensation action; and
- the person authorised to resolve an ambiguous state.
An unknown result should not be collapsed into “failed.” Give it a separate state that blocks a repeat until the external system is queried by operation identity. This is where a practical AI agent control room needs a ledger of real actions, not only a transcript of model messages.
Worked example: invoice intake without duplicate posting
Consider a hypothetical invoice-intake workflow. This is a design example, not a claim about a deployed AIEngine system. Email capture stores the original file. A model extracts fields and supporting evidence. Deterministic checks match supplier identity, purchase order, currency and totals. An authorised tool may then create a draft payable entry; payment release remains outside the model's authority.
| Condition | Ladder decision | User-visible result | Recovery evidence |
|---|---|---|---|
| Model overloaded before extraction | Retry once within the intake deadline, then rung 4 | “Received; awaiting processing” | Original file, provider error, retry timing and queue item |
| Extraction succeeds but totals do not reconcile | Rung 4, no repeat sampling | “Needs accounts review” | Extracted fields, source locations and failed rule |
| Preferred model is unavailable; evaluated read-only model is healthy | Rung 2 for extraction only | “Processed with review required” | Fallback version, evaluation scope and forced review flag |
| Draft-create call times out after dispatch | Unknown state; query by operation ID | “Confirmation pending” | Idempotency key, request ID and eventual external record ID |
| Approval or audit service is unavailable | Rung 5 before posting | “Cannot post safely; saved for later” | Control outage, preserved draft and named queue owner |
The value of this table is not the particular invoice rules. It is the separation of receipt, extraction, validation, draft creation and release. Degradation narrows authority at each boundary. It does not invent confidence by asking the model again.
Exercise the path that is meant to save you
NIST's AI Risk Management Framework says post-deployment monitoring should include override, incident response, recovery and change management, and that processes for tracking and recovering from incidents and errors should be documented. A diagram of a fallback is not evidence that it works.
Test the ladder under realistic failure combinations:
- Inject rate limits, long tail latency, malformed output and mid-stream disconnects separately.
- Remove a required source, permission service, tool response and trace sink while the model stays online.
- Replay state-changing requests with the same and different operation identifiers.
- Hold the circuit open long enough to measure queue growth, manual throughput and user behaviour.
- Recover through half-open traffic, reconcile every unknown action and verify that no reduced-mode item escaped its review flag.
Google's SRE guidance warns that rarely used degradation code is likely to fail and recommends exercising it deliberately. The same test should measure business outcomes, not only API health.
Measure useful service, not fallback activity
Track the share of requests on each rung, entry reason, time to recover and transitions that were manually overridden. Then connect those operational measures to accepted outcomes:
- completion and abandonment by rung;
- quality-rule failures and reviewer agreement;
- duplicate, unknown and compensated actions;
- oldest queue item and sustainable human throughput;
- users shown an accurate degraded-mode message;
- breaches of the end-to-end deadline; and
- cost per accepted outcome in normal and reduced modes.
Revisit the ladder after a model, prompt, corpus, tool, permission, provider, user promise or workflow change. Retire a fallback when its model version, contract, data path or human capacity is no longer dependable.
The durable objective is not to make every AI interaction return something. It is to preserve the most useful honest service the system can still support—and to stop before availability theatre becomes an irreversible mistake.
