Research-backed calculator guide

Agent Framework Overhead: Price the Runtime Before You Choose an Orchestrator

Estimate agent orchestration framework cost by separating model calls, state, queues, observability, retries, infrastructure, and engineering ownership before you choose build, buy, or hybrid.

Choosing an agent orchestration framework is not the same as choosing a model. The framework may be open source, but the production system around it still needs workers, durable state, queues, retries, logs, evaluations, security controls, and people to operate it. The right comparison is therefore not “framework license versus platform subscription.” It is the fully loaded monthly cost of delivering one successful workflow.

The practical rule is simple: price the runtime before you commit to an orchestrator. Count the model and tool calls that the workflow creates, then add the infrastructure and ownership required to make those calls reliable. Treat vendor prices as dated inputs, not permanent facts. Treat staffing, retry rates, utilization, and support as assumptions that need sensitivity testing.

What counts as framework overhead?

Framework overhead is the spend created by coordinating work rather than by producing the final answer. It has both cash and labor components.

Model and tool calls. A request may invoke a planner, a specialist, a verifier, a retrieval step, and a finalizer. Count input, cached input, and output tokens for every call. Count paid web search, code execution, file search, embedding, reranking, or external API calls separately when the provider bills them separately. OpenAI’s pricing documentation lists token rates by model and separate prices for tools such as web search, containers, file search storage, and tool calls. ^[2]

State and memory. Checkpoints, conversation history, long-term memory, vector indexes, and event records may be stored per run or per step. A framework that resumes from checkpoints can save the cost of replaying earlier calls after a failure, but the storage and write operations are not automatically free. Decide whether you retain every checkpoint, only final state, or a sampled trace.

Queues and workers. Durable execution normally needs a queue, a worker pool, or both. Capacity is driven by concurrency, execution duration, and the number of steps—not just the number of user requests. An always-on minimum worker can dominate a small pilot, while autoscaling can dominate a bursty production workload through cold starts and idle headroom.

Retries and fallbacks. A retry is a cost multiplier. If 4% of runs retry one failed step, and the retried step consumes 30% of the average model spend, retry waste is roughly 1.2% of model spend before counting the extra runtime and logging. If a fallback calls a second model after a timeout, count both attempts unless your provider bills only the completed request.

Observability and evaluation. Traces, prompt and response payloads, token counts, quality scores, retention, and human review create storage and vendor costs. They also create engineering cost when the framework does not emit the fields finance needs. Instrumentation that cannot join a run, agent, model, tool, and outcome leaves you with a total invoice but no unit economics.

People and support. Allocate engineering ownership for upgrades, on-call, security review, prompt or policy changes, evaluation maintenance, and invoice reconciliation. A build-vs-buy model that lists only cloud line items will understate the cost of an in-house runtime. Conversely, a vendor quote that includes support but excludes implementation should not be compared with a fully staffed internal estimate.

Cohere’s TCO discussion makes the same underlying point from a broader AI-ownership perspective: one user-visible answer can represent a chain of model calls, context, retrieval, tools, validation, and retries. Token price is an input to TCO, not the TCO itself. ^[5]

The cost model to use before choosing a framework

Start with a single successful workflow, then scale it to monthly volume. Define these variables:

• R: workflow runs per month.
• C: model calls per run, including planner, specialist, verifier, and finalizer calls.
• Ti and To: average input and output tokens per call, split by model where needed.
• Pi and Po: provider prices in USD per million input and output tokens.
• Q: paid tool calls per run and Pq: price per tool call.
• S: state/checkpoint writes per run and Ps: cost per write or allocated storage cost.
• W: worker and API runtime cost per month.
• O: observability, evaluation, and retention cost per month.
• F: expected retry and fallback cost per month.
• E: allocated engineering and support cost per month.

For one model tier, the monthly model cost is:

R × C × ((Ti ÷ 1,000,000 × Pi) + (To ÷ 1,000,000 × Po)).

Then add tool calls, state, runtime, observability, failures, and people. If the workflow uses multiple models, calculate each tier separately. If a provider uses different prices for cached input, cache writes, regional processing, or long context, give each category its own row. Anthropic’s current pricing documentation, for example, distinguishes base input, cache writes, cache hits, output, long-context bands, and data-residency multipliers. ^[3]

Do not bury utilization inside a vague platform line. If a worker costs $X per hour and is provisioned for H hours at U utilization, a simple allocated compute estimate is X × H ÷ U. This is not a vendor quote; it is an internal allocation assumption. Test it at 30%, 60%, and 80% utilization. Low utilization is often the reason a small managed service beats a self-hosted cluster even when the latter has a lower marginal compute rate.

For a state-machine product, also count billed transitions or execution duration. AWS Step Functions Standard Workflows, for instance, charge by state transition and charge retries as additional transitions; the AWS page shows a US East example price of $0.000025 per transition and a 4,000-transition monthly free tier. ^[4] If you use another workflow engine, substitute its documented meter. Do not transplant AWS’s number into another provider’s estimate.

Worked cost example

Suppose a US product team expects 12,000 workflow runs per month. Each run has five model calls: one planner, three specialist calls, and one verifier. The assumed average per call is 4,000 input tokens and 1,000 output tokens. Use an illustrative model rate of $2 per million input tokens and $10 per million output tokens. These are planning assumptions for the example, not a quote for an orchestration framework.

The model cost per call is (4,000 ÷ 1,000,000 × $2) + (1,000 ÷ 1,000,000 × $10) = $0.018. Five calls cost $0.09 per run, or $1,080 per month at 12,000 runs. That is the part many framework comparisons show first. It is not the complete answer.

The table below makes the remaining layers inspectable. The state and infrastructure figures are intentionally labeled assumptions so that you can replace them with your own bills or quotes.

The $3,150.96 total is an internal planning estimate, not a vendor quote. It includes no sales tax, data egress, security certification, human review, or major migration project. If the workflow has 12,000 *attempts* but only 11,500 successful outcomes, divide by 11,500 to get effective cost per successful run: about $0.274. That distinction matters when comparing frameworks because a cheaper nominal run can be more expensive per accepted outcome if it creates more retries or review work.

Now test the decision. A managed option might replace the $250 state/queue allocation and $320 runtime allocation with a quoted $900 monthly platform fee, while reducing engineering ownership from 0.10 FTE to 0.06 FTE. Under the same model, tool, observability, and retry assumptions, the managed estimate is approximately $2,950.96 per month. A self-hosted option might keep $570 of state and runtime cost but require 0.14 FTE, producing approximately $3,630.96 per month. Neither result is universal. The point is to expose which assumption creates the difference.

Run three sensitivities before signing. First, double runs while holding architecture constant; token and tool costs should scale roughly linearly, but fixed platform fees and minimum workers should not. Second, double average calls per run; this tests whether the orchestrator’s control pattern is causing cost compounding. Third, halve utilization; this tests whether your capacity plan is paying for idle workers. Keep the model rate, workload shape, and currency explicit when you compare scenarios.

Build, buy, or hybrid: compare ownership, not labels

Build can be sensible when orchestration behavior is a product differentiator, the team needs unusual state or routing semantics, and it can fund a long-lived platform owner. The price is not only initial implementation. Include upgrades, incident response, security controls, state migrations, evaluation tooling, and a replacement plan for engineers who leave.

Buy can be sensible when the workflow is standard, time-to-production matters, or the team prefers a predictable support boundary. Ask what the plan includes: model tokens, trace retention, deployment environments, concurrency, overages, data residency, support response, and export rights. A platform subscription that excludes model calls is not a fixed monthly total.

Hybrid is often the cleanest economic boundary. Keep proprietary agent logic and domain policies in your code. Rent commodity deployment, queueing, persistence, tracing, or governance where the managed service reduces operational work. The calculation should show the boundary explicitly. If a managed service includes a database but your compliance team requires a separate archive, add that archive rather than treating “managed” as “all inclusive.”

Portability is also a cost input. Record which artifacts can be exported: prompts, graph definitions, state, tool schemas, evaluations, trace IDs, and billing records. A framework with a lower first-year bill may be more expensive if migration requires replaying data or rewriting every tool adapter. Price the exit as a scenario rather than making an unsupported claim about lock-in.

To turn this into a project budget, use the AI agent development cost estimator for one-time build and staffing assumptions. Use the AI agent budget guardrail and P95 calculator to test tail spend and headroom. After launch, reconcile the estimate to provider usage with the AI API invoice reconciliation calculator. For a complete first-pass monthly model, open the AI Agent Cost Calculator | Model Full Monthly Spend.

How to keep the estimate honest

Use one dated price sheet per provider and record the currency, region, service tier, cache behavior, and whether tax is included. Official pages change. The OpenAI and Anthropic pages cited here show that model catalogs and billing categories can change, so a calculation should retain the checked date and model identifier. ^{[2] [3]}

Separate observed vendor prices from internal estimates. “$0.000025 per AWS Step Functions Standard transition in the cited US East example” is a vendor-published rate. “$320 monthly runtime allocation at 60% utilization” is your assumption. Do not place both in a table without labels.

Measure the workflow at the run level. Capture run ID, framework, graph version, model, input tokens, output tokens, cached tokens, tool calls, retry count, duration, state writes, trace retention, outcome, and human-review result. Without those dimensions, a monthly invoice cannot tell you whether cost rose because volume rose, prompts grew, retries increased, or a routing policy selected a larger model.

Finally, report a range. A base case is useful for a budget, but a low/high case is more useful for an architecture decision. Vary runs, calls per run, tokens, retry rate, utilization, and staffing independently. The goal is not to predict the invoice to the cent. It is to find the assumptions that can change the decision and instrument those assumptions before production.

FAQ

Is an open-source agent framework free?

The framework license may have no purchase price, but production use still has costs for model and tool calls, compute, state, queues, observability, security, maintenance, and support. Treat “free framework” as “no framework license line” rather than “zero total cost.”

What is the biggest hidden cost in an orchestration runtime?

There is no universal biggest line. For a small pilot it may be engineering time or an always-on worker. For a high-volume workflow it may be model calls, repeated context, retries, or a verifier loop. Calculate each layer per successful outcome and run sensitivity tests instead of assuming the platform fee dominates.

How should I compare a managed platform with a self-hosted framework?

Use the same workload and outcome definition. Compare model and tool usage, platform or compute fees, state and queue services, observability, support, engineering allocation, retry behavior, portability work, and compliance requirements. A managed quote should be dated and should state what it excludes. A self-hosted estimate should include the people who operate it.

Does adding more agents always multiply cost?

No, but it usually adds opportunities for more model calls, context transfer, tool calls, state writes, and retries. A parallel design can reduce wall-clock time while increasing total token spend. A verifier can improve acceptance rate and lower rework while increasing per-run cost. Measure cost per successful outcome, not agent count alone.

Should I include the cost of model tokens in framework TCO?

Yes. The framework may not invoice those tokens, but the orchestrated workflow causes them. Include every model call created by planning, routing, specialist work, retrieval synthesis, verification, fallback, and final response. If you compare framework-only infrastructure, label that metric separately so it is not mistaken for full workflow TCO.

How often should I revisit the estimate?

Revisit it when the model, prompt, graph, routing policy, volume, retention period, or service tier changes. A monthly reconciliation is a practical minimum for production. Recalculate before a framework migration or a major multi-agent expansion, and keep the previous assumptions so variance is explainable.