Research-backed calculator guide

AI Agent Latency SLA vs Cost: Choose the Cheapest Feasible Path

Use an AI agent latency SLA cost calculator method to compare model, tool, retry, and routing paths against p95 latency, quality, and spend constraints.

What an AI agent latency SLA should measure

An agent SLA should describe a workload, not just a model name. “The model responds in 800 ms” is incomplete when the agent may retrieve context, call a tool, retry a timeout, or ask a second model to verify the result. A useful request-level trace has at least these stages:

1. Request admission and queue wait.
2. Context retrieval and prompt assembly.
3. Every model call, including fallback and verification calls.
4. Tool execution, including external network calls.
5. Retry backoff and provider throttling waits.
6. Final response assembly and delivery.

For each stage, keep elapsed milliseconds and a correlation ID. Report p50 for the typical run and p95 for the slow tail. Use p99 when a rare but severe delay matters, such as a voice workflow or a paid transaction. Do not average stages from different requests and call that an end-to-end percentile; percentiles do not add linearly. Calculate the total duration per trace first, then calculate p50, p95, and p99 from those totals.

An SLA also needs a quality dimension. A path that stops early can look cheap and fast while producing fewer accepted outcomes. Define the task-success test before comparing paths. For a support agent, it might mean “the answer passes a reviewer rubric and the account tool was called when required.” For a coding agent, it might mean “the change passes tests and is accepted without rework.” The quality floor belongs beside latency because cost per successful task is otherwise misleading.

OpenAI’s latency guidance groups the main levers into processing tokens faster, generating fewer tokens, using fewer input tokens, making fewer requests, parallelizing, reducing perceived wait, and avoiding an LLM when a deterministic method is enough.^[1] Its cost guidance makes the same connection: reducing requests and tokens, or selecting a smaller model, can improve both cost and latency, while batch and flex processing trade lower cost for asynchronous or slower work.^[2] The calculator should expose those levers as path assumptions instead of hiding them in one “model price” field.

The feasible-path method

Use the following sequence for an AI response SLA budget.

1. Define the job and clock

Write one sentence that names the task and completion event. “Customer receives a grounded answer after one account lookup” is measurable. “Agent feels fast” is not. Choose a p95 objective and a quality floor. Example: p95 complete response ≤ 2.5 seconds and reviewed task success ≥ 98.5%.

If the product streams, record both TTFT and complete-response latency. Streaming can make the interface feel faster because output arrives incrementally, but it does not eliminate tool or generation time. Keep the two metrics separate so a fast TTFT cannot hide a slow final answer.

2. Build a complete cost meter

For each trace, add all billable model input and output tokens. Include repeated prompts, tool-result tokens, reasoning or hidden-token meters when the provider bills them, and calls made before a failure. Add tool, retrieval, queue, and infrastructure costs when those costs are material to the decision. Then report two denominators:

• Cost per attempted task = total path spend ÷ all started tasks.
• Cost per successful task = total path spend ÷ tasks that meet the quality definition.

The second figure includes failed attempts in the numerator. If a 2% failure rate triggers a human escalation, include that escalation cost when comparing the production paths. If the escalation is outside the calculation, say so explicitly.

3. Estimate before measuring, then replace estimates

A planning model can use a simple expected-cost formula:

Expected cost per attempt = (model token cost + tool cost + fixed request cost) × expected attempts + fallback cost + review cost

For a path with retry probability r, a first approximation is expected attempts = 1 ÷ (1 − r). This is only an estimate. If retries happen mainly on slow or large requests, the retry population is not independent and the simple formula will understate the p95 tail. Replace it with trace data after the workflow is running.

For latency, sum the serial stages inside each trace. Parallel tools contribute approximately the slowest branch, not the sum of all branches, plus orchestration overhead. Sequential model calls add their durations. Retry backoff is part of the user-visible clock if the user waits for it.

4. Enforce feasibility before optimizing price

A candidate is feasible only if it meets every hard constraint. A useful rule is:

feasible = (p95 complete latency ≤ SLA) AND (task success ≥ quality floor) AND (cost per successful task ≤ budget)

If one candidate fails, do not rescue it by averaging with a faster or more accurate candidate. Instead, define a routing or fallback path and model the mixture explicitly. A cheap default with an expensive escalation can be feasible, but its cost and tail must include the escalation probability.

Worked cost example

Assume a US customer-support workflow with 100,000 attempted tasks per month. The product requires complete response p95 at or below 2.5 seconds, reviewed task success of at least 98.5%, and no more than $0.01 of model spend per successful task. The figures below are planning assumptions, not vendor quotes. Token rates are shown in USD per 1 million tokens and should be replaced with the selected provider’s dated rate card. OpenAI’s published pricing page shows that input, cached input, and output are separate meters and that model rates vary materially, so one blended rate would conceal the choice being evaluated.^[3]

For the illustration, the standard model rate is assumed at $2.00 per 1M input tokens and $10.00 per 1M output tokens. The small-model rate is assumed at $0.10 per 1M input and $0.50 per 1M output. These are dated assumptions for arithmetic, not a promise that any model will deliver the listed latency or quality.

Here is the arithmetic for Path B. One attempt uses 1,200 × $2.00 ÷ 1,000,000 = $0.00240 of input cost and 300 × $10.00 ÷ 1,000,000 = $0.00300 of output cost. The base model cost is therefore $0.00540. With an estimated 3% retry rate, multiply by 1 ÷ (1 − 0.03), or about 1.0309, giving approximately $0.00557. Rounded to the table’s precision, the expected cost per attempt is $0.00556. Dividing by the 99.0% success rate gives about $0.00562 per successful task.

At 100,000 attempts, Path B’s expected model spend is approximately $556 before tools, retrieval, infrastructure, taxes, or human review. That monthly total is a scenario output, not a forecast. If the tool costs $0.001 per call, the added monthly tool spend is approximately $100 before retries; if it costs $0.01, it is approximately $1,000. Put the tool rate into the same calculator rather than treating it as immaterial.

Path A looks attractive on price and latency, but it fails the quality floor. Path C produces the best assumed quality but violates both the latency and per-success budget. Path B wins only because it is the lowest-cost candidate that passes every stated constraint. A new benchmark could reverse the choice; the method is more durable than the example’s assumed rates.

How to lower cost without breaking the SLA

First, remove unnecessary serial work. If a policy lookup and an account lookup are independent, run them concurrently and budget for the slower branch plus orchestration overhead. Do not parallelize steps that depend on one another merely to make a diagram look faster; preserve the dependency and measure it.

Second, shorten the output when the task allows it. OpenAI notes that cutting output tokens can have a much larger latency effect than cutting an equal fraction of ordinary input tokens in many workloads.^[1] Anthropic likewise recommends selecting an appropriate model, trimming prompt and output tokens, and streaming responses, while warning that hard output limits can truncate an answer.^[4] Use a structured response contract and a maximum that leaves room for a complete answer rather than an arbitrary small cap.

Third, route by task difficulty. A small model can handle classification, extraction, or a fixed-format confirmation, while a stronger model handles ambiguous or high-risk cases. The routing rule must be part of the calculation: include the classifier call, escalation probability, extra tokens, and any quality change. A smaller default is not cheaper if it creates more retries, handoffs, or failed outcomes.

Fourth, cache stable work carefully. Prompt caching or application caching can reduce repeated input processing and spend, but the hit rate, invalidation policy, and staleness risk belong in the assumptions. A cached answer that is not allowed for a current-account question is not a valid success. Mark cached and uncached paths separately in the trace.

Fifth, reserve slower lower-cost modes for work that can wait. OpenAI describes batch and flex processing as cost-saving options for asynchronous, lower-priority, or non-production workloads.^[2] They should not be mixed into an interactive p95 SLA unless the service contract explicitly permits their availability and delay behavior.

Finally, compare changes against quality and the tail. A lower p50 with a higher p95 is not an SLA improvement. A lower average cost with more tool failures may increase cost per successful task. Re-run the same task slice before and after the change, keep the metric definitions fixed, and record the date and rate-card assumptions.

Common mistakes in an AI response SLA budget

Using average latency as the SLA. An average can hide a small group of very slow traces. Calculate the percentile on complete request durations.

Stopping the clock at TTFT. TTFT is useful for streaming UX, but a complete-answer commitment needs a final-response timestamp too.

Pricing only the successful first call. Failed calls, retries, fallbacks, and human escalations consumed resources. Put them in the attempt-level ledger.

Treating token price as total task cost. Tool execution, retrieval, queues, and review may dominate a short model call.

Comparing different task mixes. A path that handles easy requests cheaply may look better when it receives fewer difficult cases. Compare the same labeled workload slice or report routing mix explicitly.

Copying a published target without a baseline. A starting point is not evidence that your agent can achieve it. Instrument first, then set a target with a stated buffer and measurement window.

For monthly planning, pair this guide with the AI Agent Cost Calculator, and use the AI agent budget guardrail and P95 calculator to test tail-spend and percentile assumptions. If the main uncertainty is model rate rather than workflow shape, open the Model pricing comparison guide and calculator. These links complement the latency-versus-cost decision instead of replacing trace data.

FAQ

What is the cheapest AI agent path that meets an SLA?

It is the candidate with the lowest expected cost per successful task after it passes every hard constraint: the defined p95 or p99 complete-response target, the minimum task-success rate, the retry and fallback policy, and any cost ceiling. A low token price alone does not qualify a path.

Should an AI agent SLA use p95 or p99 latency?

Use p95 when you want a stable measure of the slow experience that a meaningful minority of users sees. Add p99 when rare delays are operationally important, such as voice, payments, or safety actions. State the percentile, clock start, clock stop, workload slice, and measurement window in the SLA.

Does streaming satisfy a response-time SLA?

Not by itself. Streaming can lower TTFT and improve perceived responsiveness, but the complete answer may still be delayed by output generation, tool calls, moderation, or retries. Track TTFT and complete-response latency as separate metrics and choose the one that matches the contractual promise.

How do retries change AI agent cost?

Retries increase expected cost and can make the latency tail worse. A simple planning approximation is to divide first-attempt cost by one minus the retry probability. Replace that approximation with trace data when retry likelihood depends on payload size, provider load, or tool behavior.

Should cost per task or cost per successful task be the main metric?

Track both. Cost per attempted task shows operating efficiency across all starts. Cost per successful task answers what a useful outcome costs after failed attempts are included. Use the latter for path selection when candidates have different quality or completion rates.

How often should the feasible path be recalculated?

Recalculate after a model, prompt, tool, routing, retry-policy, or rate-card change, and on a regular review cadence for production. Keep the date, units, task mix, quality rubric, and assumptions with each result. A path that was feasible last quarter may not remain feasible after traffic or provider behavior changes.