Research-backed calculator guide

Latency–Throughput–Cost Frontiers for LLM Inference SLOs

Learn how to plot an LLM inference latency–throughput–cost frontier, filter configurations by SLO, and calculate cost per useful request and token.

What the frontier measures

Latency, throughput, and cost are connected but not interchangeable. TTFT covers the wait until streaming starts. TPOT describes the interval between generated output tokens. End-to-end latency is approximately TTFT plus TPOT multiplied by the number of output tokens after the first token; this makes input length, output length, and traffic mix part of the measurement rather than footnotes ^[5]. A chat SLO might constrain p95 TTFT to 250 milliseconds and p95 TPOT to 50 milliseconds. A batch workload may accept seconds of queueing but target maximum tokens per GPU-hour.

The reason a frontier exists is that prefill and decode stress different resources. Prefill processes the prompt in parallel and is generally compute-bound; decode generates tokens autoregressively and is generally constrained by memory bandwidth. Google Cloud describes this two-phase split and frames the efficient frontier as a latency-throughput curve under a fixed hardware budget ^[1]. Increasing concurrency can improve aggregate throughput through batching while worsening the latency of an individual request.

Cost is the third axis. For self-hosted inference, the meter is usually GPU or VM uptime, not a per-token invoice. If a GPU costs $g per hour and produces U useful requests per second, the simple cost per useful request is:

cost_per_useful_request = g / (U × 3,600)

Here, “useful” means a request that completed inside the agreed SLO and passed your application’s success criteria. If you care about output-token economics, divide that result by the average successful output tokens and multiply by 1,000,000. If input and output tokens have different operational costs in your model, report them separately instead of hiding the mix inside one average.

Build an SLO-filtered Pareto frontier

Start with a workload contract. Record the model and revision, precision or quantization, serving engine, GPU SKU, number of GPUs per replica, prompt-token distribution, output-token distribution, concurrency or arrival-rate test, warm-up policy, and success definition. A benchmark that changes input length between configurations can create an apparent win that is actually a workload change.

Next, choose the SLO dimensions. A practical interactive contract might be p95 TTFT ≤ 250 ms, p95 TPOT ≤ 50 ms, and an error or timeout rate below 5%. Use p95 or p99 when that is what users experience. Averages are useful for diagnosis but can conceal queueing and preemption. NVIDIA recommends examining latency-throughput curves and then selecting the highest-throughput point that remains inside the latency constraint ^[2].

Then normalize throughput. Raw generated tokens per second is not automatically useful throughput. Subtract or separately expose failed, cancelled, timed-out, or rejected work. If a replica produces 20 raw requests per second but only 92% complete successfully, its useful rate is 18.4 requests per second before headroom. If 10% of capacity is reserved for health checks, warm-up, rolling deploys, or burst absorption, apply that factor consistently to every candidate.

Finally, remove dominated points. Configuration A dominates configuration B when A is no more expensive, no slower on the SLO metrics, and at least as productive, with one strict improvement. Keep the remaining points and label the binding constraint: latency, throughput, cost, quality, memory, or capacity. A three-axis frontier can contain several valid choices; there is no universal winner without a target.

This is also why quality must be tracked when comparing quantization or other optimizations. The Inference Engineering Pareto Atlas puts quality, TTFT, TPOT, throughput, and cost per million tokens on common axes and measures anchor configurations before using simulation to explore combinations ^[4]. A speed-only frontier can recommend a configuration that fails an accuracy floor.

Worked cost example

Assume a US self-hosted service has a peak arrival target of 18 requests per second. Its SLO is p95 TTFT ≤ 250 ms and p95 TPOT ≤ 50 ms. Each successful request averages 300 input tokens and 1,100 output tokens. The numbers below are deliberately labeled assumptions, not current GPU list prices or a provider quotation.

Both points satisfy the stated p95 latency target. Configuration A looks attractive if you inspect only one replica’s hourly price, but it needs two replicas after the useful-throughput and peak-demand test. Configuration B uses the same assumed $4.80 per hour at the fleet level, serves the peak with one replica, and has the lower cost per useful request in this example. Its advantage comes from measured throughput under the same SLO, not from the fact that it has more GPUs in the abstract.

The example also shows why “cost per token” can be misleading. If your successful output average changes from 1,100 to 500 tokens, the cost per output token rises even though the GPU bill does not. If long prompts increase prefill time, the same replica may produce fewer useful requests. If the success rate drops during a burst, useful cost rises again. Keep the numerator and denominator auditable.

How to collect comparable measurements

Run a warm-up period before recording data, but report its duration and exclude it consistently. Use the same request order or a fixed randomized seed where possible. Test several concurrency levels or arrival rates rather than one “best” batch size. Record raw requests, completed requests, failures, queue time, TTFT, TPOT, end-to-end latency, output tokens, and GPU-hours.

Serving settings are part of the configuration. vLLM documents that smaller max_num_batched_tokens can improve inter-token latency while larger values can improve TTFT or throughput, and that tensor, pipeline, expert, and data parallelism have different purposes ^[3]. A point without these settings is not reproducible. Capture preemption counts too: insufficient KV-cache space can cause recomputation and worsen end-to-end latency ^[3].

Compare like with like before comparing hardware. Normalize throughput per replica or per GPU when replica sizes differ. Preserve the prompt and output distributions. Record quantization and KV-cache precision. If a system uses prefix caching, state the cache-hit rate. A cache-heavy workload can have a different frontier from a cold-prefix workload even on the same model and GPU.

For changing prices, store the currency, region, billing unit, reservation or on-demand assumption, and observation date. A GPU-hour estimate is not a quote. Add hosting, storage, networking, software, observability, and operator costs when the decision is total cost of ownership rather than marginal GPU cost. NVIDIA’s worked TCO method includes hardware depreciation, hosting, and software costs before deriving cost per prompt or token ^[2].

Common mistakes

Using mean latency for an SLO. A low average can coexist with unacceptable p95 queueing. Filter by the promised percentile.

Treating raw throughput as useful work. Retries, rejects, timeouts, warm-up, and health checks consume capacity. Put them in the denominator or show a separate utilization adjustment.

Comparing different token mixes. A 100-token response and a 2,000-token response are not the same workload. Report input and output distributions.

Assuming more GPUs always win. Tensor parallelism may reduce latency, while data-parallel replicas may improve aggregate throughput. The best choice depends on the SLO and concurrency. vLLM explicitly documents these different scaling roles ^[3].

Ignoring quality. Quantization can shift the speed-cost curve while changing answer quality. Keep the quality floor visible, as the Pareto-atlas methodology does ^[4].

Publishing a single frontier without provenance. A frontier is a measured map for a model, engine, hardware, workload, and date. It is not a timeless ranking. Re-run it after engine, model, precision, traffic, or price changes.

For a broader self-hosted cost calculation, use the self-hosted GPU inference cost calculator. For API and model-meter comparisons, see the model pricing comparison. If retrieval changes prompt length or request volume, pair this analysis with the RAG, embedding and vector database cost calculator. These tools answer different layers of the same decision rather than replacing a measured SLO benchmark.

FAQ

Is the cheapest LLM inference configuration always on the frontier?

No. A configuration can have a low hourly price but fail the p95 latency SLO, lack enough useful throughput for peak demand, or lose too much quality after quantization. It is only a frontier candidate after it passes the constraints you actually promise.

Which latency metric should an inference SLO use: TTFT or TPOT?

Use both when responses stream. TTFT controls when the user sees the first token. TPOT controls how quickly the rest of the answer arrives. For a fixed output length, you can also calculate end-to-end latency from TTFT and TPOT, but validate that approximation against measured request traces.

How do I calculate cost per useful LLM request?

Divide the fully loaded hourly fleet cost by useful requests per hour. Useful requests per hour equals raw completed throughput multiplied by the success fraction and your available-capacity factor, then multiplied by 3,600. If failures are already excluded from measured throughput, do not subtract them twice.

Does increasing batch size reduce inference cost?

It can, when additional batching raises useful throughput more than it increases required capacity or violates the latency SLO. It can also worsen TTFT or TPOT, create KV-cache pressure, and increase preemption. Sweep concurrency and batch settings under the same workload instead of assuming a monotonic benefit.

How often should an inference frontier be remeasured?

Remeasure after a model or engine upgrade, precision change, GPU or cloud SKU change, major prompt or output-mix change, routing or caching change, and material price change. For a stable production service, keep a dated benchmark record and compare the new curve with the last accepted SLO baseline.

What is the difference between a Pareto frontier and a single “best” point?

The frontier is a set of non-dominated choices. One point may minimize cost, another may maximize useful throughput inside the SLO, and another may provide extra headroom. Select a point by naming the binding business constraint rather than hiding that choice behind one composite score.