Research-backed calculator guide

LLM Inference Failover Capacity Cost: Price N+1, Warm Spares and Degrade Modes

Learn how to price LLM inference failover capacity with an auditable N+1, warm-spare, and graceful-degradation model, including a worked monthly GPU cost example.

What failover capacity means in an inference fleet

Failover capacity is reserved serving capacity for a defined failure scenario. It is different from ordinary headroom. Headroom absorbs traffic variation, deployment overlap, measurement error, or a short burst. Failover capacity absorbs the loss of something you have named and tested. Mixing the two makes a spreadsheet look safer than the service really is.

The failure domain determines the unit of the calculation:

• Replica failure: lose one model-serving replica. The remaining replicas must hold the required peak, or the router must accept a documented degrade.
• Host or node failure: lose every GPU and serving process on one machine. The surviving hosts need enough memory and throughput for the peak request shape.
• Rack or fabric failure: lose a power, network, or storage slice. The capacity model must include the affected network path, not just GPU inventory.
• Zone or site failure: lose a location. A second location must already have usable capacity, or the recovery plan must explicitly allow a slower, smaller, or read-only mode.

The arithmetic is simple, but the input is not. Inference capacity depends on request arrival rate, concurrent sequences, input and output token lengths, KV-cache behavior, batching, model precision, and latency percentiles. A model that fits in memory can still fail the SLO when a queue grows. Prime Defence’s inspected capacity guide similarly separates weight memory from KV cache, buffers, concurrency, and tested latency rather than treating parameter count as a complete GPU requirement.

A practical formula is:

Failover GPUs = max(0, required survivor capacity - capacity left after the named failure)
Provisioned GPUs = active peak GPUs + failover GPUs + separate deploy headroom

For a strict N+1 plan, the failure domain is often one host. If the host has four GPUs and the active peak needs eight GPUs, the fleet must be arranged so that losing one host still leaves eight usable GPUs. That normally means 12 GPUs across three hosts, not 10 GPUs spread in a way that leaves only six after a host loss. Placement matters as much as the total.

Your serving runtime also belongs in the capacity record. vLLM documents parallelism and engine configuration arguments that affect how a model is distributed and served; those settings are deployment inputs, not guarantees of throughput across every workload. ^[3] NVIDIA Triton documents model instance groups that can create multiple execution instances and place them on particular GPUs. ^[4] Record the exact runtime, model version, precision, parallelism, instance count, and tested request distribution beside the GPU count.

Build the cost model in four layers

1. Price the active serving pool

Start with the amount of capacity that serves the normal peak. Use a measured or load-tested requirement, such as eight GPUs, not the number of GPUs currently installed. If the workload has separate prefill and decode pools, calculate each pool independently and add them. If a single replica uses tensor parallelism across four GPUs, keep that four-GPU unit intact when testing loss; losing one GPU may remove the whole replica rather than one quarter of useful capacity.

For a cloud estimate, define an hourly rate per usable GPU-equivalent, r, and monthly hours, h. A common planning convention is 730 hours, but use your billing period if it differs. AWS states that EC2 On-Demand capacity is charged by instance-hour or second depending on instance and operating system, from launch until termination or stop; the exact rate varies by instance type and region. ^[1] Therefore, the $3 rate in the example below is an explicit planning assumption, not an AWS quote.

Active monthly cost = active GPUs × estimated GPU-hour rate × billable hours

Include attached host cost if your quote bills a whole GPU instance rather than a separable GPU. Also decide whether the estimate includes taxes, storage, inter-zone transfer, load balancing, control-plane charges, support, and reserved-commitment discounts. Do not silently compare a GPU-only number with a fully loaded host quote.

2. Add the named failover pool

A warm spare is powered and ready, or is otherwise maintained within a recovery time that your SLO permits. Its cost can be modeled as:

Warm-spare monthly cost = spare GPUs × rate × billable hours

If your provider discounts stopped capacity, use the provider’s documented billing behavior and state it. If the spare is in a second site, add the minimum host, storage, networking, replication, and monitoring resources required to make it usable. A second DNS record without serving capacity is not a failover budget.

3. Keep deploy headroom separate

If you require 20% deploy headroom on an eight-GPU active peak, round the operational requirement to two GPUs or to the next placement unit. Do not call those two GPUs failover unless they are reserved for the named failure and are still available during a deployment. AWS’s reliability guidance explicitly calls for a sufficient gap between current quotas and maximum usage to accommodate failover, which is a useful reminder that quotas and compute inventory are separate constraints. ^[2]

4. Price the degrade mode

A degrade mode changes the required capacity after a failure. Examples include accepting longer queues, lowering maximum output tokens, routing only premium traffic, disabling a nonessential retrieval step, using a smaller validated model, or switching an asynchronous job to delayed processing. The cost saving is real only if the degraded behavior is tested and acceptable to the product owner.

Record the service contract in measurable terms: maximum queue time, minimum tokens per second, allowed error rate, model-quality floor, traffic classes retained, and maximum duration. “We will degrade” is not a capacity number. It is a policy that must be translated into survivor throughput and a customer-visible limit.

Worked cost example: eight GPUs with one host of four GPUs protected

Assume a U.S. English production workload with these planning inputs:

• Tested peak requirement: 8 GPU-equivalents.
• Host shape: 4 GPUs per host.
• Named failure: loss of one host.
• Strict recovery target: survivors must still deliver the tested peak.
• Illustrative rate: $3.00 per GPU-hour, an estimate for arithmetic only, not a vendor quote.
• Billing month: 730 hours.
• Separate deploy headroom: 20% of active peak, rounded to 2 GPUs.
• Exclusions: CPU hosts, storage, data transfer, load balancer, control plane, support, taxes, software licenses, and engineering labor.

Under these assumptions, strict node-level N+1 adds $8,760 per month, or $105,120 per year, before excluded costs. If the organization accepts a degraded mode after a host loss, the required spare may be lower, but the answer depends on the measured throughput of four surviving GPUs and the traffic the product is willing to reject or delay. A four-GPU fleet that merely keeps the process alive is not equivalent to one that meets the original p95 latency target.

Now add a decision threshold. Let the probability of the named failure in a month be p, and let L be the fully measured loss from one event after considering avoided revenue, service credits, operational response, and customer impact. The warm-spare policy is economically justified by expected loss alone when:

p × L > monthly warm-spare premium + monthly operating overhead

If p = 0.08 is an internal planning assumption and the premium is $8,760, the avoided-loss threshold is $8,760 ÷ 0.08 = $109,500 per event before overhead. That is not a forecast of incidents or revenue. It is a sensitivity threshold. A team may still buy the spare below that threshold because an availability requirement, safety obligation, contractual SLO, or recovery objective makes it mandatory. Conversely, a high expected-loss estimate does not prove a particular topology will recover; the topology must pass a failure rehearsal.

Instrumentation and rehearsal are part of the cost

Count failover cost in units that expose whether the plan worked: accepted requests, rejected requests, queue time, time to first token, inter-token latency, completion time, output quality, tokens, attempts, and billed compute. OpenTelemetry defines metrics as runtime measurements and notes that metrics can support outage alerts and scheduling decisions such as scaling under demand. ^[5] Use low-cardinality dimensions such as model revision, pool, region, failure mode, and traffic class. Avoid putting raw request IDs or user IDs into every metric label; that can make telemetry itself expensive and difficult to aggregate.

A useful rehearsal removes the named domain under a representative peak trace. Drain one host, stop or isolate the serving process, or block the relevant network path. Confirm that the router detects the impairment, the spare has the correct weights and tokenizer, the quota allows recovery, and the survivors stay within the documented degrade or SLO limits. Measure total user-visible time, including detection and queueing, rather than only the time taken by the eventual successful request.

The rehearsal should also test what happens when the spare is unavailable. A plan that assumes a second site but has not tested its image, model artifact, secrets, networking, and data-residency boundary has a recovery dependency, not proven capacity. Document who can trigger the mode, which traffic is retained, how to roll back, and when the incident exits degraded service.

For a fuller baseline before adding failover, compare this model with the GPU Inference TCO Calculator. If retrieval or embedding capacity also fails over, model that pool separately with the RAG cost calculator; do not hide vector or reranker capacity inside a generic GPU percentage. For a broader delivery budget that includes staffing and run costs, see the AI agent development cost estimator.

FAQ

What is LLM inference N+1 cost?

LLM inference N+1 cost is the incremental monthly cost of one additional failure-domain unit above the minimum active fleet. If the active pool needs eight GPUs and one host contains four GPUs, strict host-level N+1 adds four usable GPUs. Multiply those four GPUs by the selected hourly rate and billable hours, then add any host, storage, networking, and operational costs. N+1 is not automatically 20%: the correct increment follows the failure-domain unit and placement constraints.

Is a GPU warm spare the same as ordinary capacity headroom?

No. Headroom covers expected variation, deployments, and scheduling uncertainty. A warm spare is reserved to absorb a named failure and should not be consumed by routine traffic or rollout overlap. A single pool can contain both, but the spreadsheet and the operational owner must keep the quantities separate. If a deploy uses the only spare, the service is no longer protected during that window.

Should the spare be powered on all the time?

Use the recovery budget to decide. A powered-off instance may lower compute spend, but model loading, image retrieval, health checks, quota allocation, and routing changes add recovery time. A cold-spare design can be valid for batch or asynchronous traffic when the allowed recovery window is long. It is not equivalent to a warm spare for interactive traffic unless the cold-start sequence has passed a representative test.

How do degrade modes reduce failover capacity cost?

A degrade mode lowers the capacity required after a failure by changing the service contract. It might keep premium requests while delaying batch work, cap output length, disable optional retrieval, or route to a smaller validated model. Measure the resulting queue, latency, error, and quality behavior. If the mode simply creates an unbounded queue or unacceptable answers, it has not reduced the required capacity; it has moved the outage into user experience.

Does a second region automatically provide inference disaster recovery capacity?

No. A second region provides a location, not necessarily loaded models, GPU quota, network paths, data access, or compatible runtime configuration. Site-level recovery needs a tested capacity reservation or a documented cold-start plan, plus a clear data-residency and routing policy. Treat the second region’s compute, storage, transfer, monitoring, and rehearsal work as cost inputs.

How often should a failover plan be tested?

Set the interval based on model-release frequency, infrastructure churn, and the recovery objective, then test after material changes. At minimum, rehearse the named failure with representative traffic and retain the measured result. The evidence should show whether the fleet recovered within the target, which requests were degraded, and whether the spare remained usable throughout the test. A spare that exists only in inventory is an assumption, not demonstrated availability.