Research-backed calculator guide

Batch Inference Window Calculator: Choose the Wait Time That Preserves Savings and Deadlines

Calculate a batch inference wait window from arrival rate, queue delay, completion yield, deadline slack, and token cost instead of treating a discount as the whole business case.

What this calculator should measure

“Batch window” can describe two different controls. A provider may expose a fixed completion window, such as OpenAI’s 24h, while your application controls how often it flushes accumulated work. Do not present a local flush interval as if it changes a provider’s contractual window. Instead, model both layers:

1. Flush interval: how long your queue waits before submission. This drives expected queue delay and batch size.
2. Provider completion window: how long the submitted job may remain in progress. This drives expiry risk and the latest possible result time.
3. Publication reserve: time for validation, retries, result ordering, and downstream writes before a human or customer deadline.

The calculator is most useful for offline classification, evaluation runs, repository embeddings, document extraction, nightly reports, and other work where nobody needs a synchronous response. Google’s batch-inference overview describes the same trade-off as high throughput and cost efficiency in exchange for higher latency and stored results rather than immediate responses. ^[5] For an interactive chatbot, the correct wait window is normally zero; route that traffic to a real-time path instead of forcing it into a batch economics exercise.

A practical input set is:

• requests arriving per hour, λ;
• candidate flush window in hours, W;
• input and output tokens per request;
• ordinary input and output rates per million tokens;
• provider batch price factor, such as 0.50 when documented;
• expected completion yield after validation and provider failures;
• p95 provider runtime and a retry or publication reserve;
• deadline from the request’s arrival or from the batch’s opening time;
• fixed orchestration, storage, and monitoring cost per job.

Use measured workload data wherever possible. A calculator output based on an assumed 98% yield is an estimate, not a vendor quote and not a promise that 98% of a future batch will complete.

The wait-window formulas

For a roughly uniform arrival stream, a request arriving just after a flush waits almost the full interval, while one arriving just before the flush waits almost nothing. The mean is therefore W ÷ 2, and the worst case before submission is close to W. If traffic arrives in bursts, use an observed arrival histogram or a conservative percentile rather than the midpoint.

Expected requests per flush are:

batch size = arrival rate × flush window

The deadline constraint should include the latest queueing case, not only the mean:

worst-case completion time ≈ flush window + p95 provider runtime + retry reserve

If your service-level deadline is measured from the first request in a batch, compare that total with the deadline. If it is measured per request, add the request’s position in the window: a request near the beginning can wait nearly W, while a request near the end waits almost zero. This is why a six-hour flush interval may have a three-hour mean wait but still expose early arrivals to nearly six hours before processing starts.

For token spend, let I and O be input and output tokens per request, and let p_i and p_o be ordinary prices per million tokens. Then:

on-demand token cost = requests × ((I ÷ 1,000,000 × p_i) + (O ÷ 1,000,000 × p_o))

batch token cost = on-demand token cost × batch price factor

cost per successful result = (batch token cost + fixed job cost + retry cost) ÷ (requests × yield)

A longer window may lower fixed cost per request because it creates fewer jobs, but it does not necessarily lower token cost if the provider applies the same batch rate to every submitted token. The window decision is therefore usually a deadline-and-reliability decision first, with a secondary orchestration-cost effect.

Worked cost example

Assume a document-tagging pipeline receives 1,000 requests per hour. Each request uses 1,500 input tokens and produces 500 output tokens. To keep the arithmetic inspectable, use illustrative on-demand rates of $2 per million input tokens and $8 per million output tokens. These rates are assumptions for the example, not a quote from any provider. Assume a documented batch factor of 0.50, a 98% completion yield, a $0.40 orchestration charge per submitted job, a two-hour p95 provider runtime, and a one-hour retry/publication reserve.

The token cost per request on the ordinary path is:

1,500 ÷ 1,000,000 × $2 + 500 ÷ 1,000,000 × $8 = $0.007

At 100,000 requests, on-demand token spend is $700. The modeled batch token spend is $350. The table compares three local flush intervals. It assumes every candidate stays within the provider’s documented job constraints and that the provider runtime estimate is valid for this workload.

The effective-cost calculation divides batch token spend plus job cost by 100,000 × 0.98 = 98,000 successful results. If the deadline is 10 hours from the first request, the 12-hour option fails even though it is marginally cheaper. The six-hour option passes the modeled 9-hour test and is the best feasible choice in this scenario. If the deadline is only six hours, the two-hour option is the only candidate that passes.

Notice what the table does not claim. It does not claim that every provider completes in two hours, that a batch discount is always exactly 50%, or that a 98% yield is guaranteed. It shows how to replace those assumptions with your own measured p95 runtime, price card, and completed-record rate. If a failed record is retried at the ordinary rate, add that retry spend instead of hiding it inside the headline discount.

How to select a window in production

Start with the deadline, not the discount. Write down when the output must be available and whether that deadline is per request, per batch, or per downstream report. Reserve time for result validation and publishing. Then calculate the maximum safe flush interval. A simple upper bound is:

maximum window = deadline − p95 provider runtime − retry/publication reserve

Use a smaller interval when arrivals are bursty, capacity is uncertain, or the cost of a late result is high. Measure p50, p95, and worst observed completion time separately. The average can look attractive while early-window arrivals regularly miss the deadline.

Next, check capacity. If a provider limits requests, input size, queued tokens, or concurrent jobs, the calculated batch size must fit those limits. OpenAI’s current documentation, for example, lists per-batch request and file limits and says unfinished requests can expire; completed requests can still be billed. ^[1] Google documents both inline and JSONL submission shapes, with the inline route intended for smaller payloads. ^[2] Bedrock’s documentation requires input and output storage in S3 and notes that batch inference is not supported for provisioned models. ^[3]

Then model yield. A batch with 100,000 submitted records and a 98% completion yield has 98,000 successful outputs, not 100,000. Track yield by model, input shape, validation error, timeout, and retry outcome. This is especially important when comparing providers: a lower nominal token rate can lose its advantage if its workflow requires more reprocessing or manual cleanup.

Finally, keep an escape hatch for urgent work. A queue that reaches its deadline reserve should flush immediately or route eligible records to a synchronous fallback. Record the reason for the fallback and its price. This lets finance distinguish a provider price change from a policy change, such as a shorter operational window during a traffic spike.

For the token arithmetic itself, compare the assumptions in the LLM cost calculator, then use the AI token cost calculator when input, output, or cached-token mix is the main uncertainty. If you are comparing several model price cards, the model pricing comparison is a better companion than copying one rate into every scenario.

Provider-specific cautions

OpenAI’s 24-hour completion window is a provider constraint, not a promise that a six-hour local flush will complete in six hours. Gemini describes a 24-hour target and says many jobs finish sooner, but actual speed depends on job size and current system load. ^{[1] [2]} Bedrock’s asynchronous S3 workflow adds storage paths and job monitoring to the model, so include those operational costs and delays rather than comparing only per-token rates. ^[3]

Price pages can also change independently from workflow documentation. AWS currently describes select Bedrock batch models as 50% below on-demand pricing, but the word “select” matters: eligibility and model-region combinations must be checked on the date of the estimate. ^[4] The same discipline applies to Google and OpenAI model availability, cached-token treatment, input modalities, quotas, and expiration behavior. Store the price date, currency, region, model snapshot, and unit basis beside each calculator result.

FAQ

Is a 24-hour batch window the same as waiting 24 hours before submitting?

No. A provider’s completion window usually describes the time allowed for the submitted job to finish. Your flush interval controls how long a request waits in your own queue before submission. If you flush every six hours into a provider job with a 24-hour completion window, an early request can face nearly six hours of queue delay plus the provider’s processing time. Model both clocks.

What is the cheapest batch inference window?

The cheapest feasible window is the longest candidate that passes your deadline, capacity, and yield constraints. In a simple model with a fixed token discount, enlarging the window mainly reduces per-job orchestration cost; it does not automatically reduce token spend. If late results trigger retries or missed-report penalties, the longer window can have a higher effective cost per successful result.

How do I calculate batch inference savings?

First calculate ordinary input and output token spend from tokens per request and published prices. Multiply the eligible portion by the provider’s documented batch factor. Add fixed job, storage, monitoring, fallback, and retry costs. Finally divide by completed results, not submitted requests. Report the result as an estimate with a checked date and units. Do not call it a guaranteed percentage saving unless the provider explicitly guarantees that treatment for your exact model and region.

How should completion yield affect the calculator?

Use successful outputs = submitted requests × yield. A 95% yield means five percent of the nominal workload is not available as a completed result. Add any retry tokens and operational handling to cost. Also test yield by window length: a longer queue may increase expiry or stale-input risk even if the provider’s token discount is unchanged.

Should interactive requests use a batch window?

Usually not. If a person or synchronous workflow is waiting, the queue delay is part of user-facing latency. Keep that path real time or use a provider mode designed for its service-level objective. Reserve batch for work that can tolerate asynchronous completion, such as evaluations, bulk extraction, offline classification, or scheduled embeddings.