Research-backed calculator guide

GPU Utilization Break-Even by Traffic Shape, Not One Average

Calculate GPU rent-versus-own break-even from peak demand, hourly traffic shape, SLO-qualified goodput, and fully loaded monthly cost instead of trusting one average utilization number.

A GPU utilization break-even calculation for LLM inference is only as good as the traffic shape behind it. An average such as “60% utilized” can hide a flat workload that needs 20 GPUs at its peak, or a bursty workload that needs 40 GPUs for a short design window. Those two services can have the same average GPU demand and opposite rent-versus-own answers.

The practical method is to calculate SLO-qualified demand for each time interval, size the owned fleet to the required peak plus stated headroom, and compare that fixed monthly cost with the rental cost of the capacity actually used. Treat utilization as a property of a trace and a capacity policy, not as a single dashboard percentage.

What “utilization” should mean here

The word utilization can refer to several different measurements. A device exporter might show compute busy time. A memory monitor might show allocated VRAM. A serving system might report running requests, waiting requests, token counters, queue time, or KV-cache usage. Those signals answer different questions.

For a cost decision, use productive, SLO-qualified capacity as the denominator. A GPU that reports high activity while requests queue beyond the product target is not delivering acceptable capacity. Conversely, a GPU can show low compute busy time because decode is memory-bound, because CPU tokenization is late, or because the traffic is intermittent. Baseten’s explanation separates compute usage, memory usage, and memory-bandwidth usage, and notes that increasing batch size can increase throughput while worsening latency. That is why “make utilization higher” is not a sufficient operating objective.

vLLM’s production metrics make the workload boundary more inspectable. Its documented endpoint exposes prompt and generation token counters, successful requests, running and waiting requests, queue time, end-to-end latency, time to first token, inter-token latency, KV-cache usage, and preemptions. ^[2] Collect those signals over the same windows as billing and traffic data. If waiting requests or queue time rise while GPU busy time is moderate, the service may be constrained by admission, CPU coordination, memory, or routing rather than raw accelerator arithmetic.

CPU-side capacity is part of the same picture. An arXiv study of multi-GPU LLM inference reports that limited CPU allocation can delay tokenization, kernel launches, and synchronization, leaving GPUs underused and increasing latency even when additional GPUs are available. ^[5] Do not classify that idle time as proof that the GPU fleet is oversized until the CPU and serving path have been checked.

The traffic-shape model

Start with a time bucket that matches the operational decision. Five-minute buckets are useful for burst response; hourly buckets are easier for a first monthly cost model. Let:

• d_t be the SLO-qualified GPU demand in bucket t.
• N_peak = max(d_t) be the peak fleet before headroom.
• N_owned = N_peak + N_headroom be the owned or reserved fleet.
• A = average(d_t) be average required GPUs across the billing window.
• U_shape = A ÷ N_owned be average demand divided by provisioned capacity.
• r_cloud be the cloud price per GPU-hour for the comparable capacity.
• r_owned be the fully loaded owned-cost equivalent per GPU-hour.
• H be paid hours in the month, such as 730 for a 365-day year divided into twelve planning months.

If cloud capacity follows demand without a minimum warm pool, the simplified monthly cloud cost is:

C_cloud = A × r_cloud × H

The owned monthly cost is:

C_owned = N_owned × r_owned × H

Set the two equal and cancel H:

A ÷ N_owned = r_owned ÷ r_cloud

Therefore:

U_break_even = r_owned ÷ r_cloud

This compact ratio is useful, but it does not remove the traffic-shape work. U_shape changes when the same average load is spread across a different peak. It also changes when you add a standby, a rollout spare, a zone-local reserve, or a minimum warm pool.

For a cloud service with a warm minimum, use a more conservative interval model instead of the simple average. In each bucket, bill max(d_t, N_min) or the provider’s actual instance count, then sum it. For a burstable serverless or spot design, add scale-up delay, interruption, and retry effects as separate assumptions. The calculator should not pretend that every provider bills an abstract GPU-demand fraction.

Worked example

The following example compares two traces with the same average demand. Both serve the same model and are assumed to achieve the same SLO-qualified goodput per GPU. The difference is how sharply demand concentrates at the peak.

Assumptions: 730 hours per month, a cloud rate of $4.50 per GPU-hour, and a fully loaded owned-cost assumption of $2.20 per GPU-hour equivalent. The owned rate is an estimate that could include depreciation or lease-equivalent cost, power and cooling, host and storage allocation, operations, and a share of platform overhead. It is not a vendor quote. AWS, for example, describes EC2 On-Demand billing as instance-hour usage from launch until termination or stop, with Linux partial hours billed per second; other providers and ownership models can use different boundaries. ^[4]

The average demand alone would miss the decision. Both traces appear to need 12 GPUs on average, so a naive monthly comparison might credit an owned fleet with 12 productive GPUs. That would ignore the 20-GPU and 40-GPU peaks that an interactive service must survive. The flat trace clears the 48.9% threshold because 12 of its 20 GPUs are needed on average. The bursty trace uses only 30% of its 40-GPU peak on average, so owning the peak fleet leaves more paid capacity idle.

Add one protected replica to each scenario and the conclusion becomes more conservative: owned capacity becomes 21 GPUs and 41 GPUs, lowering shape utilization to 57.1% and 29.3%. If the protected capacity is a standby that serves no traffic, count its cost but do not add its theoretical throughput to d_t. If it is a warm pool that can actually accept traffic within the SLO, document the failover assumption and test it.

This is also where measured goodput matters. NVIDIA recommends choosing the highest throughput that still satisfies the latency constraint, then using that throughput to estimate instances. ^[1] vLLM’s benchmark documentation supports dataset-driven tests and reports request throughput, output-token throughput, TTFT, inter-token latency, and server queue time. It also warns that cache reuse can inflate repeat-run throughput, so a benchmark trace should control warm-up and prefix-cache conditions. ^[3]

How to build a defensible trace

Start with completed requests and tokens, not with a single nvidia-smi screenshot. For every interval, retain request arrival rate, prompt-token distribution, output-token distribution, concurrency, cancellations, cache-hit behavior, and the latency objective. Translate that trace into capacity using a benchmarked goodput curve.

A useful test plan has at least three load regions: low concurrency for interactive latency, a normal sustained window, and the burst or peak window that drives fleet size. Mark a load step as passing only when the queue is stable, error rate is within target, and the required TTFT, inter-token latency, and end-to-end latency all pass. A short run that completes requests while its queue grows is not stable capacity.

Keep prefill and decode visible. Long prompts can stress prefill and tokenization, while long outputs keep decode and KV cache occupied. A workload that is “60% utilized” under short prompts may have a different goodput and memory profile under long-context requests. If a router sends different tenants or models to different replicas, calculate shape utilization per pool as well as fleet-wide. Fleet averages can hide one hot replica that breaches the SLO.

For a broader GPU cost view, use the Self-hosted GPU inference cost calculator with measured throughput and an explicit utilization assumption. If you are comparing model or provider alternatives, keep the model price and GPU capacity boundaries separate in Model pricing comparison. A retrieval-heavy agent also needs retrieval, embedding, and vector costs in the RAG, embedding and vector database cost calculator, rather than assigning every dollar to the GPU.

Limits and exclusions

This model is a decision worksheet, not a promise that owned GPUs will be cheaper. It excludes quality differences, model migration, engineering labor that is not included in r_owned, financing, taxes, egress, storage, CPU nodes, network fabric, support contracts, and the value of faster or slower capacity changes unless you add them. Cloud rates can vary by region, commitment, instance shape, interruption risk, and billing boundary. Owned hardware can have residual value, failure downtime, procurement lead time, and capacity that cannot be split cleanly.

It also assumes that the compared cloud and owned configurations deliver equivalent SLO-qualified service. If a rented GPU has a different goodput, then compare cost per completed request or cost per output token at the same latency target, not cost per nominal card. If the model cannot fit on the selected card, the utilization ratio is irrelevant. If CPU starvation or a network bottleneck leaves the accelerator idle, optimize that constraint before concluding that demand is too low.

Finally, do not annualize a single unusual peak without stating its recurrence. A launch-day spike, a weekday business-hour pattern, and a 24/7 batch queue are different traffic shapes. Run the model against a representative trace, a high-but-plausible peak scenario, and a failure or rollout scenario. Report the result as a range of assumptions, not as a universal utilization target.

FAQ

Is 70% GPU utilization the usual break-even point for LLM inference?

No universal percentage applies. The threshold is the fully loaded owned-cost equivalent divided by the comparable cloud rate. In the example above, $2.20 ÷ $4.50 gives 48.9%. A different hardware cost, cloud contract, warm pool, headroom policy, or SLO-qualified goodput changes the answer. The traffic trace still determines whether the service clears that threshold.

Why can two workloads with the same average utilization have different economics?

They may have different peaks. A flat trace that averages 12 required GPUs and peaks at 20 has 60% shape utilization. A bursty trace that averages the same 12 but peaks at 40 has 30%. If the owned fleet must cover the peak, the bursty workload pays for more idle capacity. Elastic rental can be more attractive when the provider can scale capacity close to the burst envelope.

Should raw GPU busy time be used as the utilization input?

Not by itself. Raw busy time does not prove that the service is meeting TTFT, inter-token latency, queue, or error objectives. Use serving metrics and completed work to define SLO-qualified demand, then keep raw compute, memory, bandwidth, KV-cache, and queue signals as diagnostics. vLLM documents these metrics separately through its production endpoint. ^[2]

How should I count a standby GPU in the break-even model?

Count a standby in owned cost if it is paid for, powered, or reserved. Count it in productive capacity only if the scenario assumes it serves traffic that is represented in the measured trace. A cold spare, rollout reserve, or failure replica changes the numerator of the cost comparison but should not receive fictitious tokens per second.

Does batching always improve the break-even point?

No. Batching can raise throughput and improve the amount of work per paid GPU-hour, but it can also raise queue time or first-token latency. The right batch or concurrency point is the highest measured goodput that still passes the SLO. NVIDIA describes this latency-throughput trade-off, and vLLM’s benchmark output provides the measurements needed to test it. ^{[1] [3]}

When should I use a hybrid GPU strategy?

A hybrid design is useful when baseline demand is predictable but the peak is expensive to hold continuously. Keep a dedicated or owned pool for the stable floor, then burst to cloud capacity for the tail if data, latency, and reliability constraints allow it. Model the minimum warm pool, burst duration, egress, scale-up delay, and failure behavior explicitly instead of calling the whole fleet “average utilization.”