Research-backed calculator guide

CPU vs GPU LLM Inference Cost: When Existing Servers Beat a Dedicated GPU

Compare CPU and GPU LLM inference cost per token with utilization-aware formulas, a worked break-even example, and a practical hybrid-routing decision rule.

What CPU vs GPU inference cost per token actually measures

A token-cost comparison has at least four layers.

Hourly capacity cost is what you pay to keep the deployment available. For a rented GPU, that can include the VM, attached GPU, CPU, memory, storage, and network. For an existing CPU server, it may be an allocated share of a monthly bill, depreciation, power, cooling, or simply a zero-incremental-cost scenario. These are different accounting views, so label them instead of presenting one as universally correct.

Measured throughput is the tokens-per-second result from your model, quantization, context length, prompt/output mix, runtime, and concurrency. A decode-heavy chat workload is not equivalent to a short batch job. Measure prompt tokens and generated tokens separately when possible.

Utilization converts capacity into productive output. A warm GPU that is provisioned 24 hours but generates tokens for only 20% of that time has an effective output rate near one-fifth of its active rate for an allocation-based decision. OpenCost’s inference-cost model makes this distinction explicit: usage-based cost describes active work, while allocation-based cost spreads warm hosting and shared infrastructure over actual traffic.^[1]

Quality and service constraints prevent a false bargain. A smaller CPU-served model may need more retries, longer context handling, or human review. Include those effects if they are material. Keep input and output tokens separate when their compute paths differ; prefill and decode do not have identical cost behavior.

The serving layer changes the answer too. vLLM documents continuous batching, PagedAttention, prefix caching, quantization, disaggregated prefill/decode, and support for NVIDIA/AMD GPUs and x86, ARM, and PowerPC CPUs.^[1] Those capabilities mean “CPU” and “GPU” are not complete benchmark descriptions. Record the runtime and serving configuration with every throughput number.

A practical decision rule

Choose existing CPU capacity first when all of these are true: the model fits comfortably in RAM; traffic is low, uneven, or offline; the response-time target tolerates the measured CPU latency; and the CPU cost is genuinely incremental or already committed. This is common for small quantized models, private edge deployments, scheduled jobs, and an agent’s orchestration or tool-processing layer.

Choose a dedicated GPU when the model needs GPU memory, requests arrive concurrently, output speed matters, or traffic is steady enough to keep the accelerator busy. GPUs are designed for the dense matrix operations in transformer inference, while CPUs remain valuable for request handling, tokenization, orchestration, I/O, and conditional application logic. The right architecture is often a CPU-plus-GPU system, not an attempt to put every step on one processor.

Use hybrid routing when the workload has distinguishable slices. For example, route a small classifier, cache lookup, or short extraction task to a CPU pool, and route long-context generation or high-concurrency traffic to a GPU pool. Charge the router, queue, monitoring, and fallback path to the hybrid option. If a failed CPU attempt is retried on a GPU, count the duplicate work rather than comparing only the first attempt.

Worked cost example

Assume a US deployment serving one quantized open model. The numbers below are deliberately labeled as assumptions or vendor-listed prices; they are not a promise of throughput for any particular model.

For the GPU case, use Google Cloud’s published default price for a g2-standard-4 machine with one NVIDIA L4 in the Iowa region: $0.706832276 per hour on the checked pricing page.^[2] That page states that prices are in USD and that the machine total includes the listed GPU, vCPUs, memory, and bundled components. The price is a dated vendor quote, not a universal market rate; discounts, region, storage, egress, and availability can change the result.

For the CPU case, assume an existing server allocation of $0.40 per hour for the share of CPU, RAM, power, and operations assigned to this workload. This is an analytical estimate, not a quoted provider price. Benchmark the same model and request mix at 20 generated tokens per second on CPU and 150 generated tokens per second on the L4 machine. Apply 50% CPU utilization and 20% GPU utilization to represent a light, uneven workload.

Under these assumptions, the GPU is cheaper per useful token even though it costs more per hour. It produces three times as many useful tokens per hour after utilization is applied, while its hourly cost is less than twice the CPU allocation.

Now change only the traffic shape. If the GPU is active at 5% utilization, its effective rate becomes 7.5 tokens per second and its cost becomes approximately $26.18 per million useful tokens. The CPU remains at $11.11 per million. In that case, the already-provisioned CPU is cheaper on an allocation basis. This is why a GPU “cost per active token” can look excellent in a benchmark while the same GPU is uneconomic for a small, spiky service.

The break-even utilization for the GPU in this example is easy to solve:

GPU utilization = GPU hourly cost × CPU effective tokens/hour ÷ (CPU hourly cost × GPU active tokens/hour)

Using the assumptions above: 0.706832276 × 72,000 ÷ (0.40 × 540,000) = 23.56%.

So the L4 needs roughly 23.6% allocation utilization to match the CPU’s $11.11 per million useful tokens. That threshold is not a benchmark result. It is an output of the stated rates and costs, and it must be recalculated with your measured throughput, input/output mix, and fully loaded hourly costs.

How to make the comparison fair

Start with one workload definition. Record model name and revision, parameter count, quantization, context length, input tokens per request, output tokens per request, concurrency, streaming behavior, and target latency. A CPU benchmark with a one-token prompt is not a fair proxy for a long-context agent.

Measure at several concurrency levels. At low concurrency, a GPU may be underfed. As concurrency rises, batching can improve tokens per second, but queueing and time to first token can worsen. NVIDIA recommends using metrics such as time to first token, inter-token latency, tokens per second, and requests per second, then selecting capacity subject to a latency constraint.^[3] Record the feasible operating point, not just the fastest point.

Normalize cost to the same token definition. If one report includes prompt and completion tokens while another reports completion only, the quotient is misleading. A practical blended metric is:

blended cost per 1M tokens = total hourly cost ÷ ((input tokens/hour + output tokens/hour) / 1,000,000)

For more precision, calculate separate input and output costs. Prefill, decode, KV-cache hits, and output generation can consume different resources. If your product’s margin depends on visible completion tokens, also report cost per million generated tokens and explain that it excludes prompt work.

Include idle and shared costs. Warm model memory, autoscaler headroom, monitoring, gateway CPU, storage, and engineering operations belong in the allocation view when the decision is “should we host this model?” The narrower active-compute view is useful for comparing kernels or hardware after the fleet is already paid for. Do not use the narrower number to claim that self-hosting beats an API unless the fixed costs are allocated as well.

Finally, compare like-for-like reliability. Count failed requests, timeouts, retries, fallback calls, and quality repairs. A deployment that produces 1 million nominal tokens but requires 8% duplicate generation has fewer than 1 million useful tokens. This is the point of the benchmark discipline: cost per token is an outcome of a workload and operating policy, not a permanent property of “CPU” or “GPU.”

When existing servers beat a dedicated GPU

Existing CPU capacity has an economic advantage when its incremental cost is close to zero and the alternative is a GPU that must stay warm. That advantage is strongest when model size and latency requirements allow CPU execution, traffic is intermittent, and requests can be queued or processed asynchronously. It can also be the right choice when data must remain on-premises or at the edge and a cloud GPU would introduce network, compliance, or transfer costs.

The advantage disappears when “existing” is only a procurement label. If the CPU host must be expanded, reserved exclusively, overprovisioned for peak load, or staffed with a new serving stack, allocate those costs. Also test memory bandwidth and thread scaling. More CPU threads do not guarantee more useful tokens; contention, thermal limits, and orchestration overhead can flatten throughput.

For a GPU, utilization is the main economic lever. Increase it with request pooling, continuous batching, prefix caching, quantization where quality allows, and scale-to-zero or fractional capacity where the service permits it. vLLM’s documented serving features are examples of why runtime configuration belongs in the cost model rather than in a footnote.^[1]

A good operating policy can be conditional: keep a CPU pool for low-volume and privacy-sensitive requests, send queue depth or concurrency spikes to a GPU pool, and fall back only when the latency and quality policy allows it. Recalculate the split monthly from measured useful tokens and allocation hours.

Limitations and exclusions

This example excludes model purchase or training cost, one-time engineering, taxes, network egress, persistent storage, backup, support contracts, electricity beyond the stated CPU allocation, and the cost of human review. Add them when they are material to the decision. The CPU hourly rate is an estimate; the L4 figure is a vendor-listed price observed on 2026-11-02. Neither number guarantees capacity in every US region.

The article also does not infer search volume, ranking difficulty, traffic, or revenue from the SERP. Benchmarks from another model, runtime, region, or concurrency level cannot be transplanted into this example. Re-run the measurement after model, quantization, runtime, or traffic changes.

For the deployment-side worksheet, use the Self-hosted GPU inference cost calculator. If the model choice is still open, compare API rates in the Model pricing comparison, and include retrieval costs with the RAG, embedding and vector database cost calculator. For a broader application budget, the AI agent development cost estimator helps separate build work from recurring inference spend.

FAQ

Is CPU inference cheaper than GPU inference?

It can be, but only after hourly cost, throughput, utilization, and service constraints are normalized. Existing CPU capacity often wins for small models and low or spiky traffic. A GPU can be cheaper per useful token when batching and steady utilization produce much higher throughput.

What is the simplest CPU versus GPU cost-per-token formula?

Divide the fully loaded hourly cost by effective tokens per hour, then multiply by one million: cost per 1M tokens = hourly cost ÷ (tokens/s × 3,600 × utilization) × 1,000,000. Use the same token definition on both sides and include prompt work if it is part of the workload.

Should idle GPU time be included?

Yes for a hosting or build-versus-buy decision. A warm GPU still incurs allocation cost when no request is generating tokens. You can also report active-compute cost separately, but do not compare that narrow metric with an API price or a CPU fleet cost without explaining the different bases.

When is a hybrid CPU/GPU serving architecture cheaper?

Hybrid serving is attractive when workload slices have different profiles. Send small, irregular, or orchestration-heavy tasks to CPU capacity and batchable, memory-intensive, or latency-sensitive generation to GPUs. Include routing, monitoring, queueing, fallback, and duplicate-request costs before declaring a saving.

Does a higher tokens-per-second result always mean lower cost?

No. A faster result may use a more expensive machine, violate latency targets at realistic concurrency, or be measured at an unrealistically low prompt length. Cost per useful token requires both measured throughput and the utilization at which the service operates.

How often should the model be re-benchmarked?

Re-benchmark after changing model revision, quantization, context limits, serving runtime, hardware, concurrency policy, or traffic shape. Also revisit the calculation when provider pricing or committed-capacity terms change. Keep the benchmark inputs with the cost result so a later comparison remains auditable.