Production AI Cost Is a Unit-Economics Problem
A model can answer a test question for pennies and still support an uneconomic service. The API charge is only one input to a production workflow that also retrieves data, calls tools, retries failures, waits for people, records evidence and changes over time.
The opposite claim can also be wrong: model usage is not always the smallest cost. Long contexts, high output volumes, reasoning models, voice or image generation, multi-step agents and low-utilisation self-hosted GPUs can make inference the dominant line. The correct answer is workload-specific measurement.
The FinOps Foundation’s unit-economics guidance makes a useful distinction. A resource metric such as cost per token helps engineering optimise consumption; a business metric such as cost per case resolved shows whether that consumption creates value. Production budgeting needs both.
Define a Successful Outcome Before Pricing Anything
Do not begin with “How many tokens will we use?” Begin with the unit of value.
Examples include:
- a support case resolved correctly without reopening;
- an invoice matched and posted to the system of record;
- a qualified lead accepted by sales;
- a contract clause reviewed with all required evidence;
- an employee task completed and verified;
- a customer request safely escalated within the service target.
Then define what does not count. A drafted answer that a person rewrites, a CRM update that fails silently, or a case reopened the next day is not a successful automated outcome.
Use two related measures:
| Measure | Formula | Why it matters |
|---|---|---|
| Technical unit cost | Model, retrieval, tool and platform cost / requests | Shows architecture efficiency |
| Business unit cost | Fully loaded operating cost / successful outcomes | Shows commercial viability |
A falling cost per token can coexist with a rising cost per resolved case if prompts become longer, agents make more calls, or human review increases.
If retrieval is part of the workflow, first establish what a successful answer means with the enterprise RAG evaluation guide; otherwise lower retrieval spend may simply buy poorer evidence.
Map the Full Cost Stack
A defensible total-cost model should include fixed, variable and failure-related costs.
| Layer | Costs to include | Driver to measure |
|---|---|---|
| Discovery and design | Process mapping, subject expertise, product design, legal and risk analysis | Delivery days and scope changes |
| Data and retrieval | Connectors, licences, cleaning, parsing, OCR, embeddings, indexes, storage, refresh and deletion | Documents, GB, refresh frequency, failed ingestion |
| Model and tools | Input, cached input, output, reasoning, embeddings, reranking, search, code, image, speech and other tool calls | Calls, tokens and tools per successful case |
| Integration | Authentication, APIs, queues, retries, idempotency, system-of-record verification and legacy adaptation | Systems, failure rate and reconciliation effort |
| Evaluation and assurance | Test-set creation, labels, regression runs, red-team work, impact assessment and evidence maintenance | Releases, test cases and reviewer time |
| Human operations | Approval, correction, escalation, exception handling, training and quality sampling | Review rate, minutes per case and loaded labour rate |
| Security and compliance | Access control, logging, supplier review, privacy work, monitoring and incident response | Risk tier, data classes and audit obligations |
| Reliability and observability | Hosting, telemetry, tracing, alerting, failover, support and on-call | Availability target, events and retained traces |
| Change and exit | Model migrations, prompt changes, source-system changes, provider retirement, rollback and decommissioning | Change frequency and dependency count |
AWS’s Generative AI Lens similarly treats production AI across operational excellence, security, reliability, performance efficiency, cost optimisation and sustainability—not inference alone. Microsoft’s AI workload design principles advise modelling direct and indirect financial effects, including build-versus-buy, licensing, training and operations.
Use a Formula That [Finance](/industries/finance) Can Audit
For a defined month:
Total monthly cost = amortised delivery cost + model and tool usage + retrieval and platform cost + human review and exception cost + evaluation, security and support cost + failure and recovery cost
Then:
Cost per successful outcome = total monthly cost / outcomes completed correctly under the agreed definition
Model usage can be calculated as:
Model cost = sum across models of [input tokens × input rate + cached input tokens × cached rate + output tokens × output rate + chargeable reasoning or tool usage]
Human operations can be calculated as:
Human cost = reviewed cases × average review minutes × loaded cost per minute + exception cases × recovery minutes × loaded cost per minute
Keep every input visible: volume, call count, token distribution, cache-hit rate, review rate, exception rate, latency tier, storage, support allocation and amortisation period. Record the pricing-page date and contract terms because rates and service tiers change.
A Worked Example
Consider an illustrative support workflow processing 100,000 cases a month. These are assumptions, not industry benchmarks:
| Input | Assumption |
|---|---|
| --- | ---: |
| Cases accepted automatically | 82,000 |
| Cases completed after human review | 12,000 |
| Cases returned to manual recovery | 6,000 |
| Model, retrieval and tool cost | £3,500 |
| Platform and integration operations | £6,000 |
| Evaluation, monitoring and security | £9,000 |
| Human review | 12,000 × 5 min × £0.60 = £36,000 |
| Manual recovery | 6,000 × 8 min × £0.60 = £28,800 |
| Support and change | £12,000 |
| Amortised initial delivery | £10,000 |
Total monthly cost is £105,300. If “successful AI-assisted outcome” includes the 82,000 automatic completions and 12,000 reviewed completions, the unit cost is £1.12 per successful outcome. Reporting only the £3,500 variable AI charge would imply four pence per case and conceal most of the economic system.
This example does not prove that human operations always dominate. It shows why review and failure rates belong in the same model as tokens. Change the assumptions and the result changes. Finance should be able to do that without asking engineering to rebuild the calculation.
Price the Call Graph, Not Just the First Prompt
An agentic workflow may classify a request, retrieve records, plan, call a tool, inspect the result, retry and summarise. The customer sees one answer; the provider bills several operations.
For each outcome, log:
- number of model calls by model and service tier;
- input, cached-input, reasoning and output tokens;
- retrieval and reranking operations;
- hosted or external tool calls;
- retries, timeouts and loop terminations;
- documents or records placed into context;
- end-to-end latency and success status;
- human-review and recovery minutes.
Set hard limits on agent steps, retries, output length and tool spend. An unbounded loop is both a reliability defect and a budget defect.
Use Current Cost Levers Carefully
Provider features can materially change cost, but only where the workload fits.
Batching. OpenAI’s Batch API offers a 50% discount and completes supported asynchronous work within a 24-hour window. That is useful for offline evaluation, classification, enrichment and embedding jobs—not a live customer interaction.
Prompt caching. Repeated prefixes can reduce input cost and latency. Discounts, minimum prompt sizes, expiry and retention differ by model and provider. OpenAI exposes cached-token usage and model-specific discounts in its prompt-caching documentation. Google’s documentation for Claude on Vertex AI states that cache reads are 90% cheaper than base input tokens, while cache writes carry a premium and TTL rules apply in that implementation: prompt caching on Vertex AI. Measure hit rate after deployment; do not budget as if every prompt will be cached.
Model routing. Route narrow, low-risk tasks to an evaluated smaller model and escalate harder cases. The peer-reviewed RouteLLM work shows that learned routing can trade cost against quality, but its benchmark results are not a guarantee for a private workload. A router needs its own test set, failure analysis and fallback.
Context control. Retrieve only the evidence required for the task. Long document histories increase token cost and may reduce answer quality. Measure useful-context rate, duplicate chunks and tokens per accepted answer.
Output control. Set output budgets and use a constrained schema where prose is unnecessary. Output tokens may be priced differently from input tokens and also increase latency.
Risk-based review. Review cases selected by impact, uncertainty, anomaly or policy threshold instead of sending every result to the same queue. Validate that the routing does not miss high-impact failures.
Never claim a saving until the evaluation set shows that quality, security and outcome controls remain inside tolerance.
Preserve those evaluation results, operating limits, and accepted risks in an AI assurance evidence pack so finance can trace each optimisation to current evidence.
Model Three Scenarios
A single forecast creates false precision. Build at least three:
| Variable | Expected | Stress | Optimised |
|---|---|---|---|
| Demand | Measured pilot adoption | Peak volume and rapid adoption | Routed and rate-limited demand |
| Calls per case | Observed median and p95 | Retry or agent-loop increase | Reduced call graph |
| Context and output | Pilot distribution | Larger documents and longer answers | Retrieval and output budgets |
| Human review | Observed rate | More exceptions or policy change | Lower rate proven by evaluation |
| Supplier | Current contract | Price, model or capacity change | Cache, batch or negotiated tier |
| Reliability | Normal operations | Outage, failover and manual fallback | Tested recovery path |
Use distributions, not averages alone. A small number of extremely long or looping requests can dominate spend. Forecast both monthly cost and cost per successful outcome at the median and a high percentile.
Measure Cost and Quality Together
A useful dashboard joins technical spend to workflow outcomes:
- cost per successful outcome;
- automatic completion rate;
- review and manual-recovery rates;
- p50 and p95 calls, tokens and latency per outcome;
- cache-hit rate;
- retrieval and tool cost per outcome;
- failure cost and reopened-case rate;
- quality, groundedness or policy-compliance score by model route;
- spend by use case, tenant, team and release;
- fixed-cost recovery against actual adoption.
Tag every request with a use-case identifier, customer or internal cost centre where appropriate, release, model route and final outcome. The FinOps Framework emphasises timely, accurate cost data and shared ownership between engineering, finance and business teams. A provider-level invoice without business attribution is not enough.
Treat Self-Hosting as a Different Cost Model
Comparing an API token rate with a GPU hourly rate is not a build-versus-buy analysis. For self-hosting, include hardware or reserved compute, utilisation, inference engineering, model licences, storage, networking, scaling, patching, security, observability, failover and specialist on-call coverage. Measure throughput and latency under the real prompt-length distribution, not a vendor benchmark.
Managed APIs may carry a higher visible unit rate while removing substantial platform work. Self-hosting may become attractive for steady, high-volume or sovereignty-sensitive workloads, but low utilisation can make apparently cheap capacity expensive per accepted answer. Compare both options using the same quality target and fully loaded unit cost.
Set Economic Guardrails Before Launch
Approve a range and a stop condition, not an unlimited learning period. Examples include:
- cost per successful outcome exceeds the manual baseline for two review periods without an agreed learning explanation;
- p95 calls or tokens per case breach the architecture budget;
- review or exception cost rises as coverage expands;
- required quality can only be achieved with prohibited data exposure;
- adoption remains below the level needed to recover fixed cost;
- failure and redress costs erase the measured productivity benefit;
- a provider or model change pushes the stress scenario beyond budget.
The decision may be to simplify the workflow, improve source data, constrain the agent, use deterministic automation for part of the process or stop. That is not a failed AI strategy. It is disciplined investment.
Production AI becomes financially governable when each pound can be connected to a controlled workflow and each workflow can be connected to a verified outcome. Tokens matter. The system around them decides whether they create value.
