What an AI Agent Actually Costs to Run

A prototype proves the idea works. It runs on one happy path, with clean input, on someone's laptop, for three minutes. Nothing about its cost profile carries over to production, because production is not about whether the model can answer — it is about what happens on the four hundredth call, on a bad line, with a customer who changes their mind halfway through.

The costs that matter only appear at volume: tokens per conversation, the retries nobody logged, the human handoffs at the edges, and the infrastructure that has to stay up at 3am when nobody is watching.

Inference is the number everyone budgets for, and it is usually the smaller half. The expensive parts are the ones that keep the system honest — retrieval, evaluation, monitoring, and the fallback path when the model is not confident.

A voice agent on a live phone line is a good example. Speech-to-text, the model call, text-to-speech, and the telephony leg are four separate line items, and the round trip has to finish inside a second. Cutting latency means paying for it somewhere: faster models, cached responses, or more capacity than the average load needs.

• Inference per interaction, including every retry the user never sees
• Retrieval and context assembly, which scales with how much you give the model
• Observability and evaluation, the only thing that catches silent quality drift
• Audio and telephony legs for voice, which bill by the minute regardless of quality
• Human handoff, the most expensive path and the one that decides whether the agent saves money

Every agent has a threshold below which it hands off to a person. If that threshold is too low the agent escalates constantly and you have built an expensive router. Too high and it confidently gets things wrong.

Model the fully-loaded cost of a handed-off interaction and multiply it by the escalation rate you expect. If the answer is uncomfortable, the agent is not mispriced — it is under-trained, and the fix is evaluation work rather than a smaller model.

Cost per resolved interaction is the only number that matters, and it must include the handoffs. A per-token figure looks good in a dashboard and tells you nothing about whether the system pays for itself.

Track it weekly against volume. Costs that drift upward with flat volume almost always mean retrieval is returning more context than the task needs, or the escalation rate has crept up without anyone noticing.

• Cost per resolved interaction, including escalations
• Escalation rate and the reason codes behind it
• Latency at the 95th percentile, not the average
• Context size per call, which is the most common source of quiet waste