Research-backed calculator guide

Build a Reproducible LLM Inference Cost Benchmark Workload

Define a repeatable LLM inference benchmark workload with fixed token distributions, warm-up rules, latency metrics, and cost-per-token calculations.

Why workload shape changes the answer

LLM inference has a prompt-processing phase and an autoregressive generation phase. NVIDIA defines ISL as the tokens supplied to the model, including the user request, instructions, history, and retrieved material; OSL is the number of generated tokens. Longer input tends to raise time to first token (TTFT), while longer output changes the decode work and inter-token behavior. ^[2]

That makes “the model gets 1,000 tokens” an incomplete benchmark description. It could mean 900 input and 100 output, 100 input and 900 output, or a long-tail mixture with the same average. Those cases exercise prefill, KV-cache memory, batching, and decode differently. A production-like profile should therefore record a distribution or a set of explicit slices, not only one average.

The load policy matters too. Concurrency keeps a defined number of active requests in flight; a request-rate test controls arrivals and can create a queue when the offered rate exceeds service capacity. NVIDIA’s benchmarking guidance treats concurrency, maximum batch size, request rate, sampling, and EOS behavior as parameters that can materially change measured latency and throughput. ^[2] Choose one policy for the headline comparison and record the sweep used to find the operating point.

Finally, cost is a constrained optimization, not a race to the largest throughput number. NVIDIA’s TCO method first measures latency and throughput, uses a latency requirement and peak demand to size instances, and only then derives cost per prompt or token. ^[1] A configuration that produces more tokens per second but violates a p95 TTFT or p99 request-latency target is not a cheaper production option.

Define the benchmark contract before running it

Write a small versioned workload manifest beside every result. At minimum, include these fields:

• Model identity: model name, revision or digest, tokenizer, quantization, context limit, and license or access conditions.
• Serving identity: inference engine and version, API endpoint, precision, tensor or pipeline parallelism, batching policy, speculative decoding settings, and relevant flags.
• Hardware and price basis: GPU type and count, host type, region, hourly rate, currency, whether the rate is on-demand or reserved, and whether host, storage, network, and power costs are included.
• Request shape: ISL and OSL distributions, prompt dataset or deterministic seed, system-prompt and retrieval policy, stop conditions, and maximum output tokens.
• Load shape: concurrency or arrival process, duration, request count, ramp-up, steady-state window, and failure or timeout rules.
• Measurement policy: warm-up requests, measured requests, streaming mode, metric percentile definitions, clock source, and whether client or server timestamps are used.

Do not silently change the tokenizer between candidate systems. A text prompt can become a different number of tokens across models, and even a synthetic “1,000-token” test is not equivalent if the generator or tokenizer changes. Record both requested lengths and observed token counts in the result file.

A practical profile with controlled slices

A single average profile is useful for a first pass, but three slices make the result easier to interpret:

1. Interactive chat: moderate input, short-to-medium output, streaming enabled, and a latency target for TTFT and p99 request completion.
2. RAG or long-context: large input, shorter output, explicit retrieved-document budget, and a separate cache-hit policy.
3. Generation or reasoning: smaller input, long output, a fixed output cap or EOS policy, and a decode-throughput target.

Keep the same model and deployment variables while sweeping one controlled factor at a time. For example, compare concurrency 1, 2, 4, 8, and 16 with identical requests. Then compare the three workload slices at the selected concurrency. This produces a latency-throughput curve instead of a single number that hides queueing.

Warm-up is part of the contract. A cold start can include model loading, kernel compilation, memory allocation, and cache population. It is operationally important, but mixing it into steady-state token throughput makes a serving comparison difficult to reproduce. Report cold-start time separately, send a declared number of warm-up requests, and start the measured window only after the warm-up rule is met.

Cache state also deserves an explicit label. Prefix caching can make repeated prompts look faster and more productive than a no-cache workload. The vLLM benchmark documentation warns that repeated runs can reuse prefix-cache entries and inflate throughput; it recommends varying the seed, resetting or restarting the server, or using a sweep that resets caches when cache reuse is not intended. ^[4] Therefore publish at least one cache-cold or cache-controlled result and, if caching is a production feature, a second result with the intended hit rate.

Choose metrics that answer separate questions

Use TTFT for the wait before the first streamed token. Use inter-token latency (ITL) or time per output token for the pace of subsequent tokens. Use end-to-end request latency for completion time. Use request throughput for completed requests per second and output-token throughput for generated-token capacity. Report mean or median only alongside p90 or p99; a low average can coexist with a problematic tail.

Metric names are not perfectly standardized. NVIDIA notes that tools can include different intervals and overheads in their definitions, so comparisons should preserve the measurement point and formula, not just the label. ^[2] vLLM similarly describes client-side TTFT, ITL, and TPOT and explains that TPOT divides the post-first-token time by output tokens minus one. ^[4] Put the formula and timestamp boundary in the result metadata.

A benchmark tool should emit raw artifacts as well as a summary. GenAI-Perf accepts synthetic or file-backed inputs, supports concurrency or request-rate load, records token and latency metrics, and can export CSV and JSON for later analysis. Its documentation also exposes request-shape fields such as input length, output length, timestamps, session IDs, and prefix hashes for controlled cache experiments. ^[3] The specific tool may change, but the reproducibility principle does not: save the inputs, configuration, raw responses, errors, and aggregation script.

Worked cost example

Assume one measured deployment uses four GPUs. The planning rate is $3.20 per GPU-hour, so the modeled deployment rate is 4 × $3.20 = $12.80 per hour. This rate is an explicit assumption for the example; it does not represent a quoted price from a cloud provider. The benchmark records 3 requests per second, with a mean of 900 input tokens and 300 output tokens per request. That implies:

• Input throughput: 3 × 900 = 2,700 tokens/second.
• Output throughput: 3 × 300 = 900 tokens/second.
• Combined throughput: 2,700 + 900 = 3,600 tokens/second.
• Output volume per hour: 900 × 3,600 = 3,240,000 tokens.
• Combined volume per hour: 3,600 × 3,600 = 12,960,000 tokens.

The two normalized numbers answer different questions. Output-token cost makes two decode-heavy systems easier to compare. Combined-token cost can make an input-heavy RAG system look efficient even though its TTFT and memory footprint may be higher. Publish the numerator, denominator, and token scope so a reader does not mistake one for the other.

If you are comparing managed APIs rather than GPUs, replace the deployment-rate numerator with the provider’s input, cached-input, and output rates and retain the same observed token distribution. For a mixed architecture, keep GPU serving cost, API fallback cost, storage, network, observability, and orchestration as separate lines. Do not hide retries or failed requests: show both attempted and successful request rates, then calculate cost per successful completion if reliability is part of the decision.

Turn benchmark output into a decision

Start by excluding configurations that miss the service-level objective. If the target is p99 request latency below 2 seconds, do not let a higher throughput point outside that boundary win the comparison. Next, calculate required instances from peak completed requests per second divided by the qualified per-instance throughput, rounding up and adding the stated redundancy headroom. NVIDIA uses this sequence—latency constraint, qualified throughput, then instance sizing—before its TCO calculation. ^[1]

Then compare normalized cost on the same workload slice. AWS’s published SageMaker example uses the same model, serving framework, token means, concurrency, request count, and region across GPU configurations, and reports output throughput plus average and percentile latency. It also demonstrates separate profiles for balanced chat-like traffic and long-input RAG traffic. ^[5] That is the right comparison pattern even when your tools and infrastructure differ: hold the workload constant, change the deployment variable, and preserve the profile in the report.

For broader planning, connect this benchmark to the Model pricing comparison page when an API fallback is part of the design. If retrieved context dominates the input side, use the RAG, embedding and vector database cost calculator to keep retrieval and embedding work visible rather than charging everything to generation. For agentic workflows, the AI agent development cost estimator can model the higher-level call fan-out that surrounds the serving benchmark.

Assumptions, exclusions, and limitations

This method estimates serving economics from a measured operating point. It does not predict demand, quality, rankings, traffic, or revenue. A synthetic profile can control token lengths well while failing to represent semantic complexity, tool calls, multimodal preprocessing, or real prefix reuse. A replayed production trace can be more realistic while containing private data, shifting traffic, and a distribution that is difficult to reproduce.

The worked example excludes host CPU, RAM, storage, networking, load balancer, observability, licensing, engineering, power overhead, taxes, idle capacity, autoscaling transitions, and failure retries. Add those lines when they materially affect a deployment decision. Also state whether the GPU-hour rate is a planning assumption, a list price, or an invoice-derived effective rate. Keep currencies and time periods consistent.

Do not report a precision that the experiment cannot support. If a short run produces unstable p99 values or only a handful of completed requests, label the result preliminary and increase the measured window. Re-run after model, driver, runtime, tokenizer, quantization, or traffic-shape changes. A reproducible workload is not a one-time benchmark; it is a versioned contract that makes later changes comparable.

FAQ

What inputs belong in an LLM inference benchmark workload profile?

At minimum, record model and revision, tokenizer, serving engine and version, hardware, precision, parallelism, ISL and OSL distributions, concurrency or request rate, warm-up and measured request counts, streaming mode, sampling and stop rules, cache policy, and the cost basis. Add the region, currency, endpoint timestamps, error policy, and raw-result location. These fields let another engineer reproduce the request shape instead of guessing from an average token count.

Should I benchmark fixed token lengths or real traffic?

Use both when possible. Fixed lengths are valuable for controlled sensitivity tests because one variable can change while the others stay constant. A replay of de-identified real traffic is better for a production estimate because it preserves skew, long tails, prefix reuse, and request timing. Publish the fixed-length baseline and the real-traffic profile separately; do not blend them into one unexplained average.

How many warm-up requests should I use?

There is no universal number. Choose a rule that reaches a stable serving state, record the count, and keep it identical across candidates. A practical starting point is a declared warm-up phase followed by a longer measured window, with cold-start time reported separately. If compilation or model loading is part of the user experience, measure it as its own operational metric instead of silently folding it into steady-state throughput.

Is cost per output token enough to compare inference systems?

No. It is useful for decode-heavy comparisons, but it can conceal input processing, queueing, idle capacity, and quality or latency failures. Report output-token cost alongside combined-token cost, request cost, TTFT, ITL or TPOT, p99 request latency, utilization, and successful-request rate. For a RAG workload, large inputs may dominate memory and TTFT even when output-token cost looks favorable.

How should prefix caching appear in a benchmark?

Label the cache state explicitly. Run a cache-cold or reset-cache test for a neutral baseline, then run a cache-enabled test using a measured or declared hit-rate distribution if caching is part of production. Record prefix lengths and cache hits when the tool supports them. Never compare a cache-warm run for one candidate with a cache-cold run for another.

When should I use a cost calculator instead of a benchmark?

Use a calculator for planning scenarios before you have a deployed endpoint, for API rate-card arithmetic, or for sensitivity analysis across request volume and token mix. Use a benchmark when deployment configuration, throughput, latency, utilization, or cache behavior is uncertain. In production planning, feed measured throughput and observed token distributions into the calculator, then keep assumptions visible so the estimate can be reconciled with invoices later.