Research-backed calculator guide

Speculative Decoding ROI by Acceptance Rate and Draft Cost

Calculate whether speculative decoding lowers cost per output token after acceptance rate, draft compute, GPU price, utilization, and memory overhead.

What the calculator should measure

The headline variable is acceptance rate, but it is not sufficient on its own. Acceptance rate tells you how often the target accepts proposed tokens. It does not tell you the time spent generating those proposals, the cost of loading a second model, the verification overhead, or whether the GPU was already saturated.

Start with a fixed measurement window, such as one hour or one representative batch of requests. Record the same prompt mix, output-token limit, sampling parameters, model versions, GPU type, concurrency, and serving software for both configurations. A fair comparison uses the same completed output tokens in the numerator. Do not compare “tokens generated” from one run with “requests served” from another.

The useful input set is:

• Baseline target output rate: completed output tokens per second without speculation.
• Speculative output rate: completed output tokens per second with the chosen speculator.
• Target GPU count and hourly price: the target deployment’s accelerator footprint and vendor rate.
• Draft GPU count and hourly price: include a separate draft fleet when the draft model is not colocated.
• Utilization: the fraction of paid GPU capacity assigned to this workload during the measured period.
• Acceptance rate: accepted draft tokens divided by draft tokens proposed, measured over the same traffic slice.
• Extra memory or replica cost: additional GPUs, reservations, host memory, or capacity kept warm for speculation.
• Traffic volume: output tokens per day or month, so savings can be expressed at the actual scale.

vLLM’s current documentation positions speculative decoding primarily for memory-bound, medium-to-low-QPS workloads and exposes multiple methods, including draft models, EAGLE, MTP, n-gram, suffix, and adaptive approaches ^[3]. That is a useful operational warning: a configuration that helps one interactive workload may not help a high-concurrency service whose GPUs are already compute-saturated.

The ROI formulas

There are two defensible ways to model cost, depending on how the deployment is billed.

For a dedicated self-hosted or rented deployment, use fully loaded hourly cost:

hourly cost = (target GPUs × target $/GPU-hour + draft GPUs × draft $/GPU-hour + extra fixed hourly cost) / utilization

Then convert capacity to output tokens per hour:

tokens per hour = measured output tokens per second × 3,600

Finally:

cost per million output tokens = hourly cost × 1,000,000 / tokens per hour

For a shared cluster, divide the incremental cost attributable to speculation by the incremental output tokens. This avoids charging the entire cluster to one endpoint. If speculation uses spare capacity but requires no additional GPU or memory, the incremental dollar cost can be close to zero; that does not mean the total service is free. It means the opportunity cost belongs in the capacity model rather than in the per-request invoice.

The acceptance rate is most useful as an explanatory and forecasting variable. A simple measured-throughput forecast can be written as:

expected accepted draft tokens per step = proposal length × acceptance rate

That estimate is not a complete speedup equation because acceptance usually falls deeper in a proposal window, and the target still performs verification. NVIDIA notes that accepted tokens form a prefix: after the first rejection, later proposed tokens are discarded ^[2]. Use the estimate to plan experiments, but use measured end-to-end tokens per second for the financial decision.

For ROI:

monthly savings = (baseline cost per million tokens - speculative cost per million tokens) × monthly output tokens / 1,000,000

ROI percentage = monthly savings / monthly speculative operating cost × 100

If implementation or training has a one-time cost, calculate payback separately:

payback months = one-time implementation cost / monthly savings

Do not report payback when monthly savings are zero or negative. In that case, the result is “no payback under these assumptions,” not an estimate of future revenue.

Worked cost example

Assume a team is serving a predictable code-completion workload with one target GPU. The team measures both configurations for the same prompt mix and obtains 100 baseline output tokens per second and 250 speculative output tokens per second. The speculative setup adds one small draft GPU. These are illustrative assumptions, not a vendor benchmark or quote.

For the hourly price input, this example uses a dated snapshot: Lambda’s pricing page displayed an H100 SXM self-serve row at $3.99 per GPU-hour on 2026-11-02, plus applicable sales tax or other taxes ^[4]. Treat that as a vendor-listed input for the listed configuration, not as a universal market price. The draft GPU is modeled at $1.99 per hour as an explicit assumption so the example can show draft overhead; replace it with your own measured or quoted price.

The acceptance rate is 70%, with a proposal window of five tokens. The simple planning estimate is 5 × 0.70 = 3.5 accepted draft tokens per step. That helps explain why the measured output rate increased from 100 to 250 tokens per second, but it does not replace the measurement. The target verifies proposed tokens in parallel, and the draft GPU adds $1.99 per hour before utilization adjustment.

At 10 million output tokens per day, the baseline daily cost is approximately $13.86 × 10 = $138.60. The speculative daily cost is approximately $8.31 × 10 = $83.10. The illustrative daily saving is $55.50, or about $1,665 over a 30-day month. If implementation and tuning cost $5,000, the simple payback is about three months. That payback is an estimate based on the stated traffic and pricing assumptions, not a forecast of revenue, demand, or profit.

The example also shows why “2.5x faster” is not the same as “2.5x cheaper.” Output rate is 2.5 times higher, but hourly cost rises from $4.99 to $7.48 because the draft GPU is added. Cost falls because the output-rate increase is larger than the fully loaded cost increase. With a more expensive draft model, lower utilization, or a smaller speed gain, the same acceptance rate could produce no savings.

How to run a defensible acceptance-rate test

First, define the unit of comparison. Use completed output tokens, not proposed tokens, because rejected proposals do not become user-visible output. Keep input length, output length, temperature, top-p, stop conditions, and model versions fixed. Run a baseline and speculative test over enough requests to include easy and difficult prompts. Report median and tail behavior when latency is part of the decision.

Next, capture acceptance at the request or traffic-slice level. An aggregate 70% rate can hide a 95% rate on code and a 35% rate on open-ended prose. Segment by task, tenant, language, prompt length, and concurrency. Also record mean accepted length and per-position acceptance if the runtime exposes them. vLLM documents per-request acceptance metrics and recommends reproducible offline or benchmark-CLI measurements rather than relying only on a qualitative expectation ^[3].

Then, test the proposal window. Increasing the number of speculative tokens raises the maximum gain, but later positions are less likely to be accepted. A five-token window with a 70% aggregate rate is not equivalent to five independent 70% decisions. Measure wall-clock output rate at several windows, such as 2, 4, 6, and 8, and stop increasing the window when verification or drafting overhead offsets the gain.

Finally, repeat the test at the concurrency that matters. Speculation can consume memory and scheduling headroom. If the target GPU is already saturated, adding a draft path may reduce throughput or force more replicas. A result from concurrency one should not be used as the cost model for a production service at concurrency 32 without a new measurement.

When latency improves but ROI does not

A service can become faster for each request while becoming more expensive per token. This often happens when a draft model is large, the acceptance rate is low, or the target fleet must be expanded to preserve memory headroom. It can also happen when the baseline GPU was already well utilized and speculative verification reduces the number of independent requests that fit in the batch.

Keep three outcomes separate:

1. Latency win: time per output token decreases for a request.
2. Throughput win: more completed output tokens are produced per hour of the same capacity.
3. Cost win: fully loaded dollars per completed output token decrease.

Only the third supports a cost-savings claim. The first two can still justify adoption when a latency SLO, user retention objective, or capacity constraint has a documented value. Quantify that value separately rather than treating it as an assumed dollar saving.

For the surrounding infrastructure budget, compare the result with the GPU inference TCO calculator and use the model pricing comparison when the alternative is a hosted model rather than self-hosting. If the endpoint is one part of a retrieval workflow, keep retrieval and embedding spend visible with the RAG cost calculator. For a broader product build, the AI agent development cost estimator can hold implementation work apart from recurring inference cost.

FAQ

Does a higher acceptance rate always mean a lower cost?

No. A higher acceptance rate generally makes more proposed tokens useful, but it does not price the draft model, memory, orchestration, or idle capacity. A small draft model at 65% acceptance can be cheaper than a larger draft model at 85% acceptance. Compare the fully loaded cost per completed output token and use acceptance rate to explain the result.

What acceptance rate should I target?

There is no universal target. The useful target is the lowest rate at which measured output throughput and cost meet your SLO. NVIDIA explains that higher acceptance rates can produce larger speedups and that the worst case falls back to target generation ^[2]. Start with a benchmark on your own prompt mix. A 70% rate on a predictable code slice may be valuable, while the same rate on a high-concurrency workload may not cover the draft overhead.

Does speculative decoding change model quality?

Standard rejection-sampling implementations are designed to preserve the target distribution, subject to implementation and numerical details. Google describes the method as maintaining an identical output distribution ^[1], and vLLM documents theoretical and algorithmic losslessness for its standard sampler while warning that some lossy methods deliberately relax verification ^[3]. Treat quality as a test requirement: compare task accuracy, refusal behavior, structured-output validity, and seed or numerical reproducibility for the exact runtime and configuration.

Is the draft model’s GPU cost always separate?

No. Some methods use a separate draft model, while others use a target-dependent head, n-gram lookup, or another mechanism. Separate draft GPUs are easy to model as an hourly line item. Colocated methods still have a cost through memory reservation, compute time, engineering work, or reduced batch capacity. If you cannot isolate a separate hourly price, measure the baseline and speculative deployment at the same utilization and use the observed incremental capacity cost.

How do I calculate monthly savings from acceptance rate?

Do not multiply a headline acceptance rate directly by a GPU bill. Measure or estimate completed output tokens per hour under the same workload, calculate fully loaded hourly cost, and divide cost by completed tokens. Then multiply the per-token difference by the actual monthly output-token volume. If traffic, utilization, or prompt mix changes, present a range or several scenarios instead of one false-precision result.

What should I record for an audit trail?

Record the query or traffic slice, model and draft checkpoints, runtime version, GPU type and count, hourly price source and date, utilization definition, sampling parameters, proposal window, acceptance-rate definition, completed output tokens, wall-clock duration, and whether taxes, storage, networking, engineering, and training were excluded. This makes a future price or model update an input change rather than an unexplained revision.

Limitations and exclusions

This worksheet does not forecast demand, revenue, rankings, indexing, or advertising outcomes. It does not claim that a particular acceptance rate will transfer between model families, runtimes, hardware, temperatures, or tasks. The worked example excludes engineering labor, draft-model training, storage, networking, electricity outside the quoted GPU rate, taxes beyond the vendor note, support plans, and failure or retry cost. Add those items when they are material.

The Lambda price is a dated vendor-page observation and can change by region, capacity plan, commitment term, taxes, or availability ^[4]. Replace it with the rate that applies to the deployment. Google and NVIDIA performance figures are reported examples, not a promise for this calculator or your workload ^{[1] [2]}. The search benchmark records observed result formats on 2026-11-02; the search interface did not reliably label sponsored results, so it should not be read as a guaranteed organic-ranking statement.