Research-backed calculator guide

VRAM Fit, Context Length, and GPU Count Economics

Calculate whether an LLM fits in GPU memory by precision and context length, then translate the minimum GPU count into a dated self-hosting cost scenario.

What the calculation must include

1. Weight memory by precision

Start with total parameters, not active parameters. A dense 34B model at FP16 or BF16 is approximately 68 GB of weights because each parameter uses about two bytes. INT8 is approximately 34 GB, while a nominal 4-bit format is near 17 GB before format overhead. Real files can be larger than the name suggests: scales, metadata, unquantized tensors, and alignment change the effective bytes per parameter.

Mixture-of-experts models require extra care. Active parameters describe compute for a token, but the full set of expert weights may still need to be resident. Sizing an MoE from active parameters alone can produce a false fit. Read the checkpoint’s actual tensor dtypes and file size where available, then use the effective bytes per parameter rather than an idealized label.

2. KV cache from context and concurrency

The KV cache stores attention keys and values so the model does not recompute the whole prefix for every generated token. Hugging Face documents dynamic, static, offloaded, and quantized cache strategies, each with a different memory or speed tradeoff. ^[4]

A useful approximation for a full-attention decoder is:

KV bytes = 2 × layers × KV heads × head dimension × tokens × concurrent sequences × bytes per KV element.

The first 2 represents keys and values. tokens should include the prompt plus the planned generated output, or the configured maximum sequence length when reserving worst-case capacity. Grouped-query attention can make KV heads much fewer than query heads, so use the model configuration’s num_key_value_heads; do not substitute the total attention-head count.

Cache precision is independent of weight precision. A model loaded with 4-bit weights can still use an FP16 KV cache. Conversely, an FP16 model may use FP8 or INT4 cache if the serving stack supports it and quality and latency have been tested. Hugging Face warns that quantized caching can trade memory savings for latency, while offloading can reduce GPU pressure at a throughput cost. ^[4]

3. Headroom, activations, and serving behavior

Weights and KV are not the complete runtime footprint. Prefill can create a larger temporary activation peak than decode, and the framework needs space for kernels, CUDA context, communication buffers, tokenizer or staging allocations, and allocator fragmentation. Use an explicit headroom percentage or a per-GPU reserve. A conservative first pass is 10% to 15% of the calculated working set, then validate with the actual engine.

Batch size and concurrency are not interchangeable in every engine, but both can increase active KV storage and prefill pressure. A single-user test with a short prompt may fit while a production workload with 16 concurrent sequences does not. Keep “fits in memory” separate from “meets tokens-per-second or latency targets.” NVIDIA’s H100 page lists 80 GB for H100 SXM and 94 GB for H100 NVL, with different bandwidth and interconnect characteristics. ^[1] Capacity is only one part of the deployment decision.

4. GPU count and topology

The minimum count is the smallest integer N for which the sharded per-GPU working set stays below usable memory. A simple lower bound is:

N ≥ ceiling(total working-set bytes ÷ usable bytes per GPU).

That lower bound is not a deployment decision by itself. Tensor parallelism replicates some buffers, and interconnect bandwidth affects latency. Two independent 24 GB cards are not equivalent to one 48 GB accelerator. In a tightly coupled instance, NVLink or NVSwitch may make the same count more practical than cross-host networking.

AWS’s P5 page documents a single-GPU p5.4xlarge with one H100 and an eight-GPU p5.48xlarge with 640 GB aggregate HBM3 and 900 GB/s NVSwitch connectivity. ^[2] That topology is materially different from assembling eight unrelated rental cards. The cost worksheet should therefore show both the memory count and the actual purchasable instance shape.

Worked cost example

Suppose you want to serve a 34B dense checkpoint with these planning inputs:

• 34 billion total parameters.
• 4-bit weights at 0.55 bytes per parameter, or about 18.7 GB before runtime allocations.
• 40 layers, 8 KV heads, and 128 dimensions per KV head.
• 32,768 prompt-plus-output tokens at peak.
• Four concurrent sequences.
• FP16 KV cache at 2 bytes per element.
• 12% headroom after weights and KV.
• Candidate GPU: H100 SXM with 80 GB physical memory.

The following table makes each step inspectable. Decimal GB are used for simple budgeting; the device specification and framework may report GiB instead.

On capacity alone, one H100 is sufficient for this scenario. The fit verdict is not “the model always fits”; it is “this specified checkpoint, cache precision, 32K token peak, four-sequence concurrency, and 12% reserve fit on one 80 GB H100, subject to runtime validation.” If the engine reserves 10% of the card before allocations, the usable limit is closer to 72 GB, which still leaves substantial room in this example.

Now double only the context and output budget to 65,536 tokens. KV memory doubles to 21.47 GB. The working set becomes 40.17 GB, and 12% headroom makes the planning total about 44.99 GB. It still fits on one H100 by capacity, but it consumes much more of the memory available for batching and transient peaks. If the same model uses FP16 weights instead, weight memory is about 68 GB before KV. The 32K scenario becomes approximately 88.7 GB after headroom, so the one-GPU result flips to two 80 GB GPUs by the lower-bound calculation.

For cost, use an explicitly dated rate and state the commercial assumption. AWS lists an effective Capacity Blocks rate of $5.970 per H100 accelerator-hour for the P5 entries on its pricing page, including the US East (N. Virginia) listing checked on 2026-11-02. ^[3] At continuous utilization, the simple compute-only scenario is:

$5.970 × 24 hours × 30 days = $4,298.40 per H100-month.

That is a vendor-published Capacity Blocks accelerator rate, not a universal on-demand price and not a complete invoice. Capacity Blocks have a reservation fee structure, and operating-system, storage, network, monitoring, support, taxes, idle time, and application operations can change the total. Two H100s at the same illustrative rate would be $8,596.80 per 30-day month before those additions. If your workload runs 10 hours per day for 22 business days, the compute-only estimate is instead $5.970 × 10 × 22 = $1,313.40 per H100. Use the calculator’s monthly scenario only after replacing the rate with the region, contract, reservation, or rental quote you can actually buy.

The cost consequence of context is easy to miss. At fixed weights, doubling peak tokens doubles KV allocation. That may force another GPU even though parameter count and quantization have not changed. A lower-bit KV cache, sliding-window attention, prefix reuse, or offloading can change the boundary, but each option has an effect on quality, latency, supported architectures, or throughput. Hugging Face specifically notes that offloaded caches reduce accelerator memory while potentially degrading generation throughput. ^[4]

A practical fit-to-cost workflow

First, identify the exact model revision and total parameter count. Record whether it is dense or MoE. Second, record the actual weight file’s effective precision. Third, obtain the model configuration values for layers, KV heads, and head dimension. Fourth, define a peak token budget that includes prompt, generated output, and any system or retrieved context. Fifth, multiply by peak concurrent sequences rather than average concurrency when sizing a hard capacity limit.

Next, add headroom and choose a topology. If a single high-memory GPU fits, compare its price and bandwidth with a multi-GPU option. If the model must shard, prefer an instance shape with a validated high-speed interconnect when latency matters. AWS’s P5 documentation describes one- and eight-H100 shapes and their memory totals, which is more actionable than dividing a theoretical aggregate by an unavailable card count. ^[2]

Finally, run a load test with the real serving engine. Measure peak allocated memory, time to first token, decode speed, queueing, and out-of-memory behavior at the target context and concurrency. A fit worksheet answers “can this configuration plausibly reside?” It does not answer “will this configuration satisfy my p95 latency target?” Use the Model pricing comparison when an API alternative is still in scope, and use the RAG, embedding and vector database cost calculator when retrieved context is a major part of the token and memory budget.

What the calculator excludes

This method excludes training, fine-tuning, optimizer states, gradient memory, CPU RAM requirements, disk download space, data-transfer charges, reserved-capacity commitment risk, electricity, hardware depreciation, and engineering labor. It also does not infer quality loss from quantization. A smaller memory footprint can be operationally attractive while producing a quality regression that requires more retries, larger prompts, or human review.

It also treats provider pricing as a dated scenario. Rates change by region, purchase mechanism, availability, and contract. NVIDIA’s product page supplies hardware specifications, while a cloud provider’s pricing page supplies a commercial rate under its own terms. Do not combine a vendor quote for one GPU shape with a different topology and call the result a confirmed monthly bill.

FAQ

How much VRAM does a 7B model need at 4-bit?

A rough weights-only estimate is 3.5 to 4.5 GB, depending on the file’s effective bytes per parameter. Add KV cache, activations, runtime reserve, and concurrency. A 7B model with a long context can need materially more than its compressed weight file, so “4-bit 7B” is not a complete VRAM answer.

Does quantization reduce KV-cache memory?

Not automatically. Weight precision and KV-cache precision are separate choices. Loading 4-bit weights can leave an FP16 cache unless the runtime is configured otherwise. Quantizing the cache may reduce memory, but test latency and output quality for the model and workload. Hugging Face documents quantized and offloaded cache strategies as explicit configuration choices. ^[4]

How do I calculate GPU count for a long context?

Calculate weights, then KV cache at the peak token budget and concurrency, then activations and headroom. Divide that planning total by usable memory per GPU and round up. After that lower bound, check the available instance topology, replicated buffers, interconnect, and engine behavior. The rounded-up number is a starting point, not proof of throughput.

Is total parameter count or active parameter count used for VRAM?

Use total resident parameters for weight memory unless the runtime demonstrably pages or offloads weights. Active parameters mainly describe per-token compute in an MoE model. If all experts are loaded, sizing from active parameters alone underestimates memory.

What is the difference between VRAM fit and inference cost?

VRAM fit is a capacity test. Inference cost adds the price of the GPU-hours, minimum replica count, utilization, idle periods, storage, networking, software, and operations. A configuration can fit on one GPU but cost more per successful request if it cannot meet latency or concurrency targets and needs additional replicas.

Should I use a cloud instance’s total GPU memory as one pool?

Only when the serving stack and topology support the required sharding method. AWS documents P5 instances with one H100 or eight H100s connected through NVSwitch, but that does not make arbitrary GPUs in separate hosts one unified pool. Validate tensor-parallel or pipeline-parallel support, communication bandwidth, and per-GPU allocation before committing.