Research-backed calculator guide

Cost versus latency routing: build a Pareto frontier instead of picking the cheapest provider

Learn how to build an LLM routing cost-latency Pareto frontier using normalized token cost, p95 latency, quality floors, and a worked traffic-mix example.

What a Pareto frontier means for LLM routing

Suppose a candidate route has lower cost, lower p95 latency, and equal or higher quality than another route on the same task set. The second route is dominated and should not receive production traffic unless a missing constraint changes the comparison. If one route is cheaper but slower, or faster but less accurate, neither dominates the other. Both belong on the frontier.

The comparison must use one fixed evaluation slice. Do not compare a short classification prompt on one provider with a long tool-using agent trace on another. Split traffic into meaningful segments such as extraction, customer support, coding, and long-context research. Record input tokens, output tokens, cache state, retries, status codes, tool use, and a quality outcome for each request. OpenTelemetry’s LLM observability guidance recommends capturing request metadata, model identity, token usage, cost, and request duration because those values explain both spend and performance changes. ^[9]

Use the vendor rate card for the price inputs, but do not confuse the rate card with the realized request cost. OpenAI publishes separate input, output, and cached-input rates per million tokens. ^[1] Anthropic similarly separates base input, cache writes, cache reads, and output, and notes that tokenization can vary between model families. ^[2] Google’s Gemini pricing also separates input, output, and context caching, with some models charging thinking tokens as output. ^[3] Those categories should be represented in your ledger before a router compares providers.

A simple fully loaded cost estimate is:

cost_request = input_tokens × input_rate + cached_input_tokens × cached_rate + output_tokens × output_rate + tool_costs + expected_retry_cost + router_cost

Each token term is divided by 1,000,000 when rates are quoted in USD per million tokens. If a request can escalate from a cheap model to a stronger model, add the escalation probability multiplied by the second call’s expected cost. If a request is retried, add the retry probability multiplied by the retry cost. These are estimates until you replace them with trace data.

Build the frontier in five steps

1. Define hard constraints before optimization

Start with constraints that are not preferences. Examples include a minimum answer-pass rate of 93%, a p95 latency limit of 2.5 seconds, a required JSON schema, a US-only processing rule, or a maximum request cost of $0.003. Filter candidates that fail these requirements. A weighted score can hide a fatal failure; a hard quality or compliance floor should not be traded away for a small price improvement.

2. Normalize the request economics

For each route, use the same expected input and output tokens. Track cache-hit and cache-write categories separately. Include model-specific tokenization in the measured token counts rather than assuming that 10,000 characters become the same number of billable tokens everywhere. Include gateway fees, routing classification, tool calls, retries, and regional or priority-tier multipliers when they apply. OpenRouter documents that its provider routing considers effective price, throughput, time to first token, rate limits, reliability, and tool-call quality, rather than list price alone. ^[4]

3. Measure latency as a distribution

At minimum, store time to first token (TTFT), streaming duration, total duration, and p50/p95/p99 by segment. A single average can look excellent while a long tail violates an interactive SLO. OpenRouter describes TTFT as network transit plus queue wait and prompt prefill, while streaming time depends on decode speed. ^[5] A router should therefore keep TTFT and total completion latency as separate fields, even if the policy ultimately uses total p95.

4. Score quality on the work the route will serve

Use a labeled evaluation set with the same prompt shapes, tools, context lengths, and output checks that occur in production. A response can be fast and cheap yet fail a required field. Store a pass/fail or calibrated score, then report confidence intervals when two candidates are close. Segment-level quality matters: one provider may be excellent for extraction and weak for code repair.

5. Remove dominated candidates and allocate traffic

For a minimization axis such as cost or latency, lower is better. For quality, higher is better. Candidate A dominates B when A is no more expensive, no slower at the chosen percentile, and no worse in quality, with a strict improvement somewhere. Delete B from the default choice set. Allocate among the survivors using explicit business rules: a quality-first route may accept higher cost, while an interactive route may require a p95 ceiling.

Do not let a fallback silently change the economics. OpenRouter notes that a failed initial provider attempt adds latency to that request, while its routing system can route around providers after observed errors or 429s. ^[5] Record the initial attempt and the fallback attempt separately so that the blended cost and tail latency remain visible.

Worked cost example

The following is an illustrative worksheet, not a vendor quote. Assume 100,000 requests in a month. Each request contains 2,000 input tokens and produces 500 output tokens. Ignore caching and tool charges in this first pass. The assumed rates are deliberately labeled inputs so you can replace them with current rate-card values. Use the model pricing comparison calculator to test a current provider mix. For token-count assumptions, cross-check the AI token cost calculator; for a monthly request baseline, compare it with the LLM cost calculator.

For Route A, the token calculation is (2,000 × $0.40 + 500 × $1.60) / 1,000,000 = $0.0016 per request. For Route B it is (2,000 × $2.00 + 500 × $8.00) / 1,000,000 = $0.0080. The illustrative mix costs 0.60 × $0.0004 + 0.30 × $0.0016 + 0.10 × $0.0080 + $0.00005 = $0.00137 per request, or $137 for 100,000 requests, before retries and other charges.

The arithmetic is not permission to route 60% of traffic to Route C. Route C fails the stated quality floor, so that share is valid only for a different segment whose measured quality clears the floor. If the whole workload needs at least 93% quality, remove C and recalculate the mix. With a 70% A and 30% B mix, the cost is 0.70 × $0.0016 + 0.30 × $0.0080 + $0.00005 = $0.00293 per request, or $293 for 100,000 requests. That is a transparent trade-off: stronger quality headroom costs more than the low-cost route.

The p95 result also needs care. A 60/30/10 mix does not produce a p95 of the weighted average of 1.4, 1.9, and 3.2 seconds. The slow tail can occupy the 95th percentile. Calculate p95 from request-level observations after routing, and report the p95 of each segment and the blended workload. If Route B is reserved for complex requests, report the complex-segment SLO separately rather than hiding it inside an overall average.

Price speed without losing the cache

Routing can create a hidden latency and cost tax when it switches providers between turns. A multi-turn agent may need to send a long context again, lose a warm KV cache, or trigger a second prefill. Research on dynamic quality-latency routing explicitly models communication, computation, router, generation, and context-switching costs. ^[8] Preserve stable prompt prefixes where possible, and treat cache warmth as a measured state rather than a promise.

Service tiers are another source of frontier movement. OpenRouter documents default, flex, and priority tiers as different capacity and price trade-offs, and says billing follows the tier that actually serves the request. ^[6] A “fast” tier is not automatically a winner: compare its realized p95 and total cost against standard or flex candidates on the same workload. Re-run the frontier when prices, models, tokenizers, capacity, or traffic composition changes.

What to log after launch

A useful route ledger has one row per attempt, not just one row per user request. Include request ID, task segment, candidate model, provider, service tier, input and output token counts, cached token counts, quoted rates, estimated cost, router decision time, TTFT, total duration, status code, retry number, fallback flag, quality result, and policy version. Langfuse’s analytics guidance similarly recommends slicing volume, cost, latency, quality, and errors by use case, model, chain step, time, version, geography, and language. ^[10] When actual invoices arrive, reconcile the estimate against the AI API invoice reconciliation calculator.

Review the frontier weekly at first. Watch for a route that was non-dominated in a benchmark but becomes dominated under peak-hour queueing, longer prompts, or a changed cache rate. OpenTelemetry’s guidance also highlights rate limits as a production signal, not merely an operational nuisance. ^[9] A 429 can increase cost and latency through retries even when the nominal token rate is unchanged.

FAQ

Is the cheapest LLM provider always the best route?

No. The cheapest rate can fail a quality floor, produce a bad p95 tail, lack a required tool feature, or incur more retries. Compare fully loaded request cost and measured quality and latency for the same task segment. The cheapest eligible route is a useful baseline, not an automatic production decision.

What is the difference between a Pareto frontier and a weighted score?

A frontier removes candidates that are strictly worse across all chosen axes. A weighted score ranks the remaining candidates using preferences such as “quality is twice as important as cost.” Use constraints and dominance removal first. A weighted score can then choose among non-dominated candidates, but it should not hide a route that violates a hard requirement.

Should routing use p50 or p95 latency?

Use both, but use the percentile that matches the product SLO for the decision. p50 describes the typical request; p95 exposes the slow tail that many interactive products feel. Keep TTFT and total duration separate. A route can start streaming quickly but finish slowly, or wait in a queue before generating quickly.

How do retries change the frontier?

Retries increase both expected cost and tail latency. Add retry_probability × retry_cost to expected request cost, and measure the duration of retried requests separately. A provider with a lower nominal price can lose after 429s or transient failures. Keep failover logic distinct from normal cost or quality optimization so an outage policy does not distort the baseline comparison.

How many requests do I need for a reliable frontier?

There is no universal number. Use a representative, versioned sample large enough to include normal traffic and important rare failures, then quantify uncertainty for close comparisons. Stratify by task, input length, tool use, and context length. If two candidates are separated by less than the measurement noise, leave both on the frontier and collect more evidence instead of forcing a false winner.

Can a router optimize provider choice and model choice at the same time?

Yes, but keep the candidate identity explicit. A candidate should be a complete route such as model plus provider plus service tier, region, and relevant cache policy. Changing any of those can change price, queueing, reliability, or data handling. Compare complete routes, not only model names copied from a pricing page.