Research-backed calculator guide

Right-Size an LLM GPU Cluster from Concurrency and p95 SLO

A practical method to size LLM GPU replicas from peak concurrency, token shape, measured goodput, p95 latency targets, and explicit headroom.

If you are doing LLM GPU capacity planning concurrency SLO work, do not start with a rule such as “one GPU serves 20 users.” Start with the workload and a measured service curve. The defensible sequence is: prove that the model and its live KV cache fit, measure the highest sustainable per-replica goodput while the p95 latency target holds, calculate peak replicas, and then add separately stated capacity for failure, rolling deployment, burst absorption, and routing skew.

That distinction matters because concurrency is not the same as throughput. Ten long-context requests can consume more memory than 50 short requests, while the same request rate can produce different queueing behavior when prompts and completions change. A GPU’s utilization percentage also does not prove that a latency SLO is safe. The number you need is the highest completed request rate at which your latency and error objectives still hold for a production-shaped trace.

Separate memory fit from concurrency capacity

The first question is whether one replica can hold the model and the active requests without relying on an accidental near-OOM condition. For a dense FP16 model, the rough weight estimate is parameter count multiplied by two bytes, but it is only a starting point. Quantization metadata, replicated tensors, sharding imbalance, CUDA graphs, kernels, activations, allocator fragmentation, and host or device reservations all change the real footprint.

KV cache is the part that makes concurrent users expensive. A live request keeps key/value state for retained prompt tokens and generated tokens, so a long context or a long-running generation can occupy memory while other requests wait. Page-based allocation can reduce fragmentation and share reusable prefixes, but it does not create unlimited memory. The PagedAttention paper describes near-zero KV-cache waste and flexible sharing in vLLM, with measured throughput improvements in its evaluated configurations; those are research results, not a promise for your model or hardware. ^[4]

A useful per-replica worksheet has these rows:

• Reported GPU memory, in GB.
• Weight memory after the exact serving format is loaded.
• KV-cache allocation for the joint prompt/output workload.
• Runtime, graph, communication, and temporary-buffer memory.
• Measured operational reserve for variance, upgrades, and admission control.

Subtract the last four categories from reported memory to obtain a conservative KV budget. Do not assume the product-page total is fully available. NVIDIA’s nvidia-smi documentation notes that ECC can reduce available framebuffer memory and that the driver may reserve memory for internal use; its memory accounting can also differ on systems where the operating system manages framebuffer memory. ^[5]

The H100 specification is a useful illustration of why units matter: NVIDIA lists 80 GB of GPU memory and 3.35 TB/s of memory bandwidth for H100 SXM, with up to seven 10 GB MIG instances. ^[3] Those are hardware specifications, not a claim that any particular model will achieve a particular requests-per-second result. If you use MIG, each slice is a different capacity and performance experiment.

Turn concurrency and p95 into a measured goodput

A capacity test should recreate the work users create, not just send identical short prompts. Record at least the prompt-token distribution, generated-token distribution, cache-hit ratio, cancellation behavior, tool pauses, burst window, and the maximum admitted sequence length. If reasoning mode is optional, include its share and its longer output distribution in a separate scenario.

vLLM’s benchmark CLI supports a finite request rate, burstiness, and a maximum concurrency limit. The documentation describes finite-rate load as a way to simulate realistic request timing, while --max-concurrency models backpressure at a client or gateway. It also documents ramp-up testing for finding maximum throughput under a latency budget. ^[1] A practical test plan is:

1. Warm the model and runtime, then exclude the warm-up window from results.
2. Replay the same production-shaped trace at several offered rates and concurrency limits.
3. Hold each step long enough to reveal a growing queue rather than mistaking a short burst for stable capacity.
4. Capture p95 TTFT, p95 inter-token latency, p95 end-to-end latency, errors, timeouts, and completed requests.
5. Mark a run as passing only when every declared objective passes and the queue is stable.
6. Repeat from a fresh process and with one replica removed for a failure scenario.

Define per-replica goodput as the highest completed request rate that passes all those gates. If 6.2 requests per second completes during a test but the waiting queue grows throughout the window, 6.2 is not sustainable goodput. It is an unstable observation.

The vLLM metrics endpoint exposes the signals needed to explain why a run passed or failed. Its documented metrics include running and waiting requests, KV-cache usage, preemptions, prompt and generation tokens, request queue time, end-to-end latency, inter-token latency, and time to first token. ^[2] Use those signals to distinguish a memory-admission limit from a decode-throughput limit or an API and tokenizer bottleneck.

Worked cost example

The following is a planning example, not a vendor quote or a measured claim about H100 performance. It shows how to connect concurrency, a p95 target, explicit capacity decisions, and monthly GPU spend.

The memory check runs alongside the service test. Suppose the selected replica uses a 13-billion-parameter FP16 checkpoint. The illustrative weight estimate is 13B × 2 bytes = 26 GB. If measurement allocates 10 GB to runtime and temporary buffers and reserves 8 GB for operational headroom, the provisional KV budget is 80 − 26 − 10 − 8 = 36 GB on an 80 GB H100 SXM specification. ^[3] If an instrumented run measures 0.22 MB of KV storage per live token and the admission target is 2,048 live tokens per request, the rough page-free estimate is 0.22 MB × 2,048 ≈ 0.45 GB/request, or about 80 theoretical equal-length requests before additional effects. That is not the allowed production concurrency: page rounding, mixed lengths, temporary peaks, queue policy, and p95 behavior require a lower tested admission limit.

The $2.50 per GPU-hour figure is an explicit planning assumption. It is not a current provider price, and it excludes CPU nodes, RAM, storage, networking, egress, orchestration, support, electricity, taxes, and engineering labor. Replace it with the actual quote or invoice rate for your region and contract. If the real rate is $3.20, the same five-GPU run is $3.20 × 5 × 730 = $11,680/month before those excluded costs. For a fuller comparison, enter the GPU count and rate into the Self-hosted GPU inference cost calculator, then reconcile the estimate with the workload trace rather than treating utilization as a cost allocation.

Choose the right headroom, not a magic percentage

Headroom is a policy decision. A single spare replica may be enough for a small, stateless service with fast replacement and admission control. A multi-zone service may need one spare per failure domain. A service with a slow model load may need warm capacity before the queue grows. State which risk each addition covers:

• Failure or maintenance: capacity that remains after losing one replica.
• Burst absorption: capacity needed while autoscaling reacts to a short peak.
• Rollout: capacity for a canary and old/new versions at once.
• Traffic skew: capacity for a hot tenant, sticky session, or route that is not evenly balanced.

Do not call quota capacity. A cloud quota may allow you to request GPUs while regional inventory, placement, networking, or model-serving limits still prevent the deployment. Measure readiness time, and test the degraded case with one replica unavailable. The right decision might be a warm pool, a lower admission limit, request shedding for background traffic, or a different topology—not simply more GPUs.

For a broader financial view, compare the cluster’s GPU line item with model and provider alternatives in Model pricing comparison. If the service is part of a retrieval-augmented agent, include retrieval and vector costs with the RAG, embedding and vector database cost calculator rather than allocating every dollar to the GPU.

FAQs

How many GPUs do I need for a given number of concurrent users?

There is no reliable user-to-GPU conversion without a request profile. Concurrent users become arrival rate, prompt length, output length, cache behavior, and latency requirements. Use the peak request rate and measured per-replica goodput for the exact model and runtime. A user who holds a long streaming response can consume a different amount of KV memory from a user who sends a short request.

Should I size to average latency or p95 latency?

Size to the p95 (or stricter percentile) specified by the product SLO. Average latency can look healthy while a queue, long prompt, or decode-heavy tail breaches the experience for a meaningful set of requests. Track TTFT, inter-token latency, and end-to-end latency separately because a service can meet one and miss another.

Can GPU utilization tell me whether I need another replica?

No. Utilization is a diagnostic signal, not the capacity decision. A service can show moderate utilization while KV memory, queueing, interconnect, or a single hot route limits p95 latency. Add a replica when the measured goodput curve stops passing the declared SLO, then verify the new replica actually receives balanced traffic.

Does MIG make a GPU equivalent to several smaller GPUs?

No. MIG provides partitioned capacity and isolation, but each profile has its own memory and compute limits. NVIDIA lists up to seven 10 GB MIG instances for H100 SXM, but a MIG slice will not reproduce the service curve of a full 80 GB GPU. Benchmark the model, batch shape, and SLO on the selected profile. ^[3]

How often should I rerun an LLM capacity plan?

Rerun after changing the model revision, quantization, context limit, serving runtime, GPU type, prompt/output distribution, caching policy, or SLO. Also rerun when production metrics show a sustained shift in p95 queue time, KV usage, preemptions, or token mix. Keep the benchmark inputs and software versions with the result so a later comparison is meaningful.

What should I do when the model fits but p95 still fails?

First identify the phase and queue: prefill, decode, KV admission, host preprocessing, network streaming, or routing. Then test the smallest change that addresses that bottleneck—admission control, batching, prompt limits, a different parallel layout, more replicas, or a separate prefill/decode pool. Do not infer that a larger GPU will solve a bottleneck you have not measured.