Research-backed calculator guide

Scale-to-Zero vs a Warm GPU Pool for Interactive Inference

Compare scale-to-zero, a warm GPU floor, and scheduled prewarming with a transparent cost-per-latency-compliant-request model for interactive inference.

What “scale to zero” changes in the cost model

A warm replica charges for readiness. If one GPU remains allocated for a 30-day, 24-hour month, its floor is 720 GPU-hours before it serves a request. The formula is:

warm_floor_cost = GPUs × warm_hours × GPU-hour_rate

Scale-to-zero removes that continuous floor, but it does not make startup free. The first request after a quiet period may wait for scheduling, a container, model weights, runtime initialization, and inference warmup. Some platforms bill that startup time; others have a different request or minimum-duration meter. Confirm the billing contract before treating cold-start seconds as billable GPU seconds.

For a provider-neutral estimate, model cold wakeups explicitly:

scale_zero_cost = GPU-hour_rate × (request_GPU_seconds + cold_wakeup_GPU_seconds) ÷ 3,600

where cold_wakeup_GPU_seconds = cold_wakeups × cold_start_seconds × GPUs. Add CPU, memory, storage, network egress, gateway, logging, and retry charges separately when they are material. Do not mix a provider’s per-request quote with a GPU-hour calculation unless the meters are normalized first.

The latency denominator matters just as much. If 1,000 of 20,000 monthly requests arrive after a scale-down event and miss a two-second first-token target, the useful denominator is 19,000 compliant requests, not 20,000 total requests. This is why “cost per request” can hide a capacity decision. Report both total cost and cost per compliant request.

Three deployment choices

Scale-to-zero

Choose this when idle periods are long, traffic is genuinely sporadic, and the caller can queue or accept a cold-start response. Batch inference, internal tools, development environments, and asynchronous workflows are common fits. A request queue can absorb startup time, but the queue itself needs a timeout and a retry policy.

AWS’s documented pattern makes the operational trade explicit: an endpoint may scale in to zero to save cost, but after reaching zero it cannot answer until at least one instance is provisioned. AWS also warns that invocations during provisioning produce an error and that the process can take several minutes ^[1]. Treat that behavior as a requirement to design around, not as an incidental implementation detail.

Warm floor

A warm floor keeps one or more replicas ready continuously. It is appropriate when a human is waiting, first-token latency is part of the product promise, or a short burst must be served before autoscaling can add capacity. The floor should be sized from the traffic that can arrive during a measured startup window, not from average daily requests.

A warm floor is not the same as peak capacity. Keep enough warm capacity for the latency-critical baseline, then let autoscaling add replicas for longer bursts if the queue and startup window permit it. If a GPU can serve 30 concurrent requests under the target latency, one warm GPU may be enough for the steady slice; if the burst arrives faster than new replicas become ready, reserve more.

Scheduled prewarming

Scheduled prewarming keeps capacity warm during known peaks, such as business hours, and returns to zero during predictable quiet windows. Its cost is:

scheduled_cost = GPUs × scheduled_hours × GPU-hour_rate + residual_cold_start_cost

It works only when the schedule matches reality. Include a safety buffer for early arrivals, late sessions, time-zone variation, and unexpected demand. If the product is global, a single eight-hour window may understate the real warm requirement.

Modal documents the relevant controls in concrete terms: scaledown_window reduces the chance of a subsequent cold start but bills resources while they remain idle; min_containers prevents the function from scaling to zero; and buffer_containers adds ready capacity for bursts ^[3]. These are useful abstractions even if your provider uses different names.

Worked cost example

Assume one GPU serves an interactive model. The following inputs are intentionally simple so you can replace them with production measurements:

• Monthly requests: 20,000 requests.
• GPU time per request: 2 seconds.
• GPU rate: $2.00 per GPU-hour. This is an illustrative assumption, not a vendor quote.
• Cold starts: 1,000 per month.
• Cold-start duration: 30 seconds on one GPU.
• Month length: 720 hours.
• Latency target: the request must avoid the cold-start path to count as compliant.
• Scheduled window: 8 hours per day for 30 days, with all 20,000 requests inside that window.

The raw inference work is 20,000 × 2 = 40,000 GPU-seconds, or 11.11 GPU-hours. At $2.00 per GPU-hour, that active work costs $22.22.

The scale-to-zero wakeup work is 1,000 × 30 = 30,000 GPU-seconds, or 8.33 GPU-hours. That adds $16.67, for a modeled monthly total of $38.89. If every cold-start request misses the target, only 19,000 requests are compliant. Cost per compliant request is $38.89 ÷ 19,000 = $0.00205, or about 0.205 cents.

A one-GPU warm floor costs 1 × 720 × $2.00 = $1,440 for the month. The active request work is inside those already-paid GPU-hours, so do not add it a second time to the floor calculation. If all 20,000 requests meet the target, cost per compliant request is $1,440 ÷ 20,000 = $0.072, or 7.2 cents.

An eight-hour daily prewarm costs 1 × 240 × $2.00 = $480. Add the same $22.22 active-work estimate if your billing meter charges active GPU time separately from the scheduled reservation. The modeled total is $502.22, or $0.02511 per compliant request, equal to about 2.51 cents. This example assumes all requests land inside the window; requests outside it need their own cold-start or overflow line.

The table is a decision aid, not a provider invoice. If cold-start time is not billed, scale-to-zero’s GPU line falls, but the latency and retry lines do not automatically disappear. If the warm GPU can serve other workloads, allocate only its incremental share to this endpoint. If requests queue rather than fail, record queue delay and the percentage that still meets the end-to-end target.

Finding the break-even point

For a simplified comparison, let r be GPU dollars per hour, g the number of GPUs, H the monthly hours, N requests, t GPU seconds per request, W cold wakeups, and c cold-start seconds.

The scale-to-zero estimate is:

C0 = r × g × (N × t + W × c) ÷ 3,600

The always-warm estimate is:

Cwarm = r × g × H

Ignoring non-GPU charges, scale-to-zero is cheaper when:

N × t + W × c < 3,600 × H

The interesting variable is often W, not N. Two workloads can process the same 20,000 requests but have very different numbers of traffic gaps. A single daytime burst may have few wakeups; a chat service with thousands of short sessions may repeatedly pay startup overhead. Measure the distribution of gaps longer than the platform’s scale-down window. Do not substitute average requests per hour for that distribution.

The financial break-even can also be latency-adjusted. Assign a defensible penalty p to each non-compliant request, such as a measured retry cost, support cost, or contribution-margin loss. Then compare:

risk_adjusted_cost = infrastructure_cost + (non_compliant_requests × p)

The penalty is an internal estimate, not a vendor price. Keep it as a separate sensitivity variable. If no defensible value exists, report infrastructure cost and compliance rate side by side instead of inventing a business value.

How to size a warm pool from the traffic trace

Start with the latency target and the startup window. Measure from the scale-up decision to the point at which the replica can serve the target workload, including model load, CUDA or engine initialization, health checks, and routing. A platform’s container boot number is not automatically an inference-ready number.

Next, calculate demand that can arrive during that window. If the endpoint can receive 120 requests during a 30-second startup and one GPU can process only 60 of them within the target, the warm floor or queue policy must cover the shortfall. For a multi-GPU model, count the full replica rather than treating each GPU as an independent request worker.

Then segment by route. Keep interactive chat, voice, or coding traffic on the strictest latency budget. Let batch, evaluation, and asynchronous document work use scale-to-zero. A single global warm setting often overpays for workloads that do not share the same user-visible SLO.

Finally, test a scheduled policy against actual gaps. Compare the cost of extending a warm window by 15 minutes with the expected cold-start and retry cost in that interval. Modal’s documentation illustrates why this is a real trade: a longer idle window can prevent a cold start, but the resources used while idle are still billed ^[3].

For broader infrastructure context, use the Self-hosted GPU inference cost calculator and compare the result with the Model pricing comparison. If inference is part of a retrieval pipeline, also separate model serving from the RAG, embedding and vector database cost calculator, because vector and embedding charges do not belong in the GPU warm-pool denominator.

Practical limitations and exclusions

The worksheet does not predict provider availability, queue placement, or a particular p95. It assumes the GPU rate, startup time, and request duration are stable enough for a scenario. Real systems vary with model size, quantization, image layers, weight-cache locality, concurrency, GPU type, region, and capacity pressure.

It also excludes taxes, storage, egress, observability, control-plane charges, CPU and RAM reservations, gateway fees, failed-request charges, retries, replicas for high availability, and the engineering cost of operating a scheduler. Add each line if it is material. A warm pool can be cheaper after shared allocation, reserved capacity, or multi-tenant utilization; it can be more expensive after redundancy and idle memory are included.

Run at least three traffic traces: a quiet overnight gap, a normal weekday, and a burst or incident day. Report total cost, compliant-request rate, p95 or p99 first-token latency, and the count of cold wakeups. This makes the decision auditable and prevents a low average GPU bill from hiding an unacceptable interactive experience.

FAQ

Is scale-to-zero always the cheapest option for GPU inference?

No. It is often cheapest on raw idle capacity for sparse traffic, but cold starts, retries, queue time, and missed latency targets can change the effective result. Compare cost per latency-compliant request rather than GPU-hours alone. For sustained traffic, a persistent GPU may be cheaper because the serverless premium and repeated startup behavior are paid frequently ^[2].

What is the warm pool GPU cost?

For a simple floor, multiply the number of GPUs by warm hours by the GPU-hour rate. One GPU kept warm for 720 hours at an assumed $2.00 per GPU-hour costs $1,440 before other charges. Replace the illustrative rate with the rate and billing unit in your contract. If the GPU serves other routes, allocate its shared cost rather than charging the full floor to one endpoint.

How do I estimate a scale-to-zero LLM cold start?

Measure from the first request after scale-down until the model is genuinely ready to serve. Include scheduling, image or volume access, weight loading, runtime initialization, engine compilation, health checks, and routing. Record a distribution, not only an average: p50, p95, and worst observed cold starts are more useful for an interactive SLO.

When does a warm pool beat serverless GPU inference?

A warm pool becomes attractive when the value of predictable latency and the cost of cold-start consequences exceed the idle GPU floor. Operationally, it is a strong candidate when the service is user-facing, traffic gaps are short, startup time is long relative to the SLO, or demand can spike before autoscaling adds capacity. Quantitatively, include a penalty for non-compliant requests only when you can support that estimate.

Is scheduled prewarming better than keeping one GPU warm all month?

It can be, when peak windows are predictable and most requests occur during them. In the example, eight hours per day costs 240 warm hours rather than 720. The policy fails if users arrive outside the window or if a global audience shifts the peak. Add a buffer and measure outside-window cold wakeups before treating the schedule as production-safe.

Does a shorter scaledown window eliminate GPU inference cost?

No. A shorter window may reduce idle hours, but it increases the chance that the next request starts cold. A longer window may improve latency while charging for idle resources. Provider controls differ, so model the observed gap distribution and actual billing unit. The decision is about total cost and compliance, not the timeout value in isolation ^[3].