The Agentic AI Cost Problem: Why Token Spend Is Not the Unit That Matters

Published: at 03:50 AMSuggest Changes

The bill arrives before the business case has caught up. A CIO sees a manageable model invoice, the operations lead sees agents completing work, and finance asks whether the programme is under control. All three can be right for a month and still be wrong about the economics.

Token spend is easy to report because providers meter it cleanly. Agentic work is harder. A single request can retrieve policy documents, invoke several systems, retry a failed call, hand a case to a reviewer, and leave an exception for the next shift. The model charge is visible. The cost of finishing the work safely is scattered across the operating model.

That distinction matters once an agent touches a customer, a case queue, a production system or a decision with financial consequence. Leaders who manage only the model bill will optimise the cheapest component and miss the expensive behaviour around it. The unit that deserves management attention is the completed, controlled business task.

A token price is an input price

Tokens still matter. Managed-model services commonly charge different rates for input, output and cached input. Choice of model, response length, context window and routing policy all affect spend. OpenAI’s prompt-caching guidance describes how a reusable matching prefix can avoid repeat processing of earlier context, reducing cost and latency for suitable workloads. OpenAI

Those are useful levers. They are not a service economics model.

Take two procurement-support requests. The first agent retrieves the current policy, checks the supplier record, produces a compliant recommendation and closes the task. The second consumes a similar number of tokens, pulls an outdated policy, fails twice against a supplier API and sends the buyer a correction request. The provider invoice will show two modest runs. The procurement function has seen one completed task and one new piece of work.

The opposite problem also occurs. A longer context can be good value if it avoids a faulty change plan, a second customer contact or an escalation to a scarce specialist. Token reduction that strips out approved evidence produces a cheaper answer and a costlier process. A mature team treats context as an input to task quality, then measures whether the extra input earns its place.

Choose a denominator that represents the work. For a service desk, it might be cost per correctly resolved request. For finance, cost per evidence pack accepted without rework. For operations, cost per compliant change plan or triaged alert with no missed high-severity case. A drafting assistant can use an approved draft as its unit. An agent that takes action should be judged on a task completed within its authority, quality and control limits.

This changes who needs to be in the room. Engineering can explain model usage. Finance can allocate spend. The process owner knows whether work was accepted, corrected or abandoned. None can describe the economics alone.

Follow one task from trigger to closure

The fastest way to expose the gap is to trace a representative task from trigger to final disposition. Include the happy path, the slow path and the cases that end with a person taking over. Teams often find the model call is a modest line item surrounded by costs they have never joined together.

Start with inference and context. Capture input, output, cached and reasoning tokens where the provider exposes them. Record the model, region, price version and every substantial source of context: system instructions, retrieved material, tool definitions and conversation history. Do not merge model tiers into a single opaque total. A routing change can move difficult work onto a more expensive model long before anyone notices it in a monthly average.

Then account for retrieval and data access. Search, embeddings, vector-store operations, document processing, database queries and API calls all carry a direct charge or consume shared capacity. Quality belongs in this ledger as well. Retrieval that returns stale, irrelevant or unauthorised material creates two costs: wasted inference and the human effort needed to correct the result.

Tool and orchestration costs deserve the same treatment. A run may generate queue activity, serverless execution, browser sessions, workflow-engine steps, API calls and third-party transaction fees. In a workflow that creates a case in a licensed SaaS product or calls a paid data service, tools can outweigh model spend at the margin.

Reliability overhead is where weak designs become expensive. Record retries, timeouts, fallback routes and duplicated calls. Categorise the cause: a transient provider failure, a tool error, missing data, an ambiguous instruction or an unsafe request. These are different management problems. Adding another retry may help with a transient failure; it merely burns money when the agent lacks the right data or authority.

Human assurance is part of the run cost, not an embarrassing exception to it. Approval, sampling, correction, escalation and incident response all consume capacity. AWS guidance on agentic AI economics makes the right point: human intervention belongs where the cost of failure is higher than the cost of review, and that effort must be counted. AWS The aim is a deliberate assurance model, with people focused on the decisions where their judgement protects value.

Finally, allocate the shared platform cost. Observability, evaluation, identity, security controls, integration maintenance and service operations support every production workflow. Avoid charging an entire enterprise platform to its first use case; that turns early experiments into false failures. Avoid the opposite fiction as well. Once a pilot is a service, the controls and people that keep it safe are part of its economic reality.

Each run should create a traceable record tagged with workflow, business owner, environment, model configuration, action class and outcome. Link it to the ticket, case or transaction when appropriate. That join between technical telemetry and business outcome turns cost attribution into evidence.

Monthly averages hide the expensive work

A monthly AI bill is a finance statement. It is rarely a useful operational diagnostic.

A service desk agent might resolve common password or policy queries cheaply while using extensive retrieval and tool retries for ambiguous requests. An operations agent might have a low median cost and a damaging tail during an upstream outage. Averages blur both patterns, which is why a programme can look efficient until volume rises or a dependency fails.

Track four views at a minimum:

Segment these measures by workflow type, model route, data source, customer or operational tier, and exception class. Leaders should be able to distinguish a price increase caused by growth in demand from one caused by longer context, poorer retrieval, a new paid tool, a routing change or a rise in difficult cases.

FinOps provides useful foundations for this. The FinOps Foundation’s 2025 framework widened its Cloud+ scope beyond conventional cloud cost data, while FOCUS 1.2 added allocation and invoice-reconciliation detail across SaaS and PaaS reporting. FinOps Foundation FOCUS 1.2 That makes AI spend easier to see and allocate. It does not establish that an agent did work the business accepted.

Product and operational telemetry supply the missing evidence. Finance can show what was paid. Engineering can show what ran. The accountable workflow owner must show whether the result met the promised standard and altered the measure that justified the investment.

Put the control at the point of action

A budget alert after month-end is a report on an incident that has already happened. Agentic systems need economic controls inside the workflow.

Set an action budget for each agent: maximum model calls, tool calls, records touched, retries, elapsed time and estimated marginal spend before it stops or escalates. Set lower limits where the task is uncertain, externally facing or high impact. These thresholds are policy decisions, agreed with the business owner and risk function, rather than arbitrary engineering settings.

The agent also needs safe failure paths. When approved context is unavailable, it should abstain or seek help. When a tool fails twice, it should create an exception containing the evidence already gathered instead of looping. When the work is predictable, a deterministic workflow may be the better design. Anthropic’s guidance on effective agents recommends starting with the simplest solution and recognises the latency and cost trade-off of agentic systems. Anthropic Treat that trade-off as an architecture choice, not a prompt-tuning detail.

Routing is another control point. A smaller model may handle classification, extraction or routine routing; a larger model can be reserved for ambiguous synthesis and higher-risk judgement. Test the policy against representative cases and include correction effort in the result. A route that saves on inference while doubling reviewer time is a poor commercial decision.

Context hygiene matters for the same reason. Keep stable instructions and tool definitions reusable where caching supports it. Retrieve narrowly. Remove obsolete material. Cap unproductive conversation history. Record when the agent lacks enough evidence to proceed. The target is sufficient, current context with a clear purpose for every significant token.

Make task economics an operating discipline

Agent economics need named decision rights. The business owner owns the value hypothesis and the acceptable outcome. Product or operations owns workflow measures and exception design. Engineering and platform teams own instrumentation, routing, limits and integration reliability. Finance and FinOps own allocation, visibility and forecasting. Risk and security decide where assurance outweighs speed.

That does not require a new central committee. It requires a short, regular operating review while the workflow is new, then a cadence matched to its rate of change and materiality. Review cost, quality, exceptions and business outcome together. Decide whether the next move is tuning, redesign, expansion, containment or retirement.

The leadership test is simple and demanding: choose one production agent and ask for the cost of its last 100 accepted tasks, the exceptions that consumed the most money, and the person authorised to change its budget or stop it. If management cannot answer within a working session, it does not yet control the economics of that agent. It controls a token bill.

Sources

Previous Post
When Agents Change Production: The New SRE Playbook for Autonomous Work
Next Post
AI That Cannot Be Tested Cannot Be Trusted: Building an Enterprise Evaluation System