Research-backed calculator guide

Partial Stream API Billing: Record Cost When a Generation Is Cancelled or Disconnected

Learn how to record and reconcile API cost when a streamed generation is cancelled, disconnected, or missing its final usage event.

Why cancellation does not answer the billing question

A client disconnect describes what your application received. It does not, by itself, describe what the model service generated or what the provider charged. A stop button may reach the upstream request quickly, slowly, or not at all. A reverse proxy may close the downstream connection while the upstream model call continues. A provider may bill already generated tokens, bill a completed upstream call, or expose a delayed usage record. Those are different events and need different ledger fields.

The missing-final-event problem is documented in the OpenAI Chat Completions reference. When streaming usage is requested, the usage object is null except for the last chunk, which contains the request’s token statistics; the same reference warns that an interrupted or cancelled stream may not deliver that final usage chunk ^[1]. The safe conclusion is not “the request cost nothing.” It is “the local stream response is incomplete.”

Anthropic’s streaming documentation shows a different event shape. A stream begins with message_start, emits content-block events, can emit message_delta events, and ends with message_stop. Its documentation also warns that the token counts in message_delta.usage are cumulative ^[2]. That makes event parsing and reconciliation provider-specific. A generic ledger should preserve the raw event type and the provider protocol rather than assuming that every final event, delta, or chunk means the same thing.

The four cost states to use

A useful interrupted-stream record has a cost_status that can move forward but should not erase earlier evidence.

Exact means the provider returned authoritative billable usage for the request. This might arrive in a normal terminal response, a delayed retrieval endpoint, an activity export, or another provider record that identifies the request. Store the evidence location and the timestamp at which it was observed.

Estimated means your application has calculated a plausible amount, but at least one billable field is inferred. For example, input tokens may be known while output tokens are estimated from accumulated text with a documented tokenizer or heuristic. Store the estimator version, the fields estimated, and the pricing snapshot used. An estimate is useful for budgets and customer-facing provisional reporting, but it is not a provider quote.

Unknown means the request reached a stage where work may have occurred, but you lack enough evidence to calculate a defensible amount. Keep the request visible in spend monitoring with an unknown amount or an explicit range. Unknown is more honest than zero when a disconnected stream has no final usage and no reliable provider record.

Reconciled means a previous estimate or unknown has been replaced or annotated with authoritative provider evidence. Do not overwrite the original record. Store estimated_cost, actual_cost, variance, reconciled_at, and the provider evidence identifier so finance and engineering can explain why a past dashboard changed.

These states also prevent a common downstream error: using the UI outcome as the billing outcome. A user may see a partial answer and rate it as unsuccessful, while the provider may have generated and charged for more output than reached the browser. Conversely, a gateway may cancel upstream work early and have only a partial, estimated amount. “The customer did not receive a complete answer” is a product result, not a universal billing rule.

Build the interrupted-stream ledger

At minimum, persist one durable row per upstream attempt. Use a stable request_id and a separate attempt_id if your application retries. Include parent_request_id when one user action can create multiple provider calls. This stops a retry from being mistaken for a continuation of the original stream.

Capture the provider and resolved model, not only the model alias requested by the application. Add the API protocol, endpoint, service tier or deployment where relevant, and a pricing snapshot date. Rates can change, and a later calculation using today’s rate can create a false variance. The Google Cloud Vertex AI pricing page, for example, presents model-specific input and output token prices and separate cached-token categories ^[3]. Record the applicable row and unit instead of storing one undifferentiated “token price.”

For usage, keep separate fields for input_tokens, output_tokens, cached input or cache-write tokens, reasoning tokens, tool-call or retrieval units, and any media-specific meter. Each field should carry an evidence state such as provider_reported, locally_counted, estimated, or missing. A single total_tokens field hides exactly the distinction that matters after an interruption.

For stream behavior, record stream_started_at, first_chunk_at, last_chunk_at, terminal_event_at, and disconnect_at. Store termination_reason values such as user_cancelled, client_disconnect, proxy_timeout, provider_error, or unknown. Also record whether cancellation was sent upstream and whether the upstream connection was confirmed closed. A downstream HTTP 499 or an AbortController event is evidence about the client path, not proof of upstream cancellation.

Finally, save enough raw evidence to reproduce the calculation without saving secrets: provider request ID, response ID where available, HTTP status, error type, event count, accumulated text length, usage payloads, and a redacted event sample. Never put API keys, authorization headers, or customer-private prompts into a reconciliation export.

Calculate a lower bound before estimating

Start with fields you can defend. If input usage was reported before the stream broke, use it as an exact input component. If a terminal or in-band usage field reported output tokens before cancellation, preserve it as provider-reported output. If all you have is text received by the client, count it only with a documented tokenizer or approximation and label the result accordingly.

The lower-bound calculation is not necessarily the final bill. It is the amount supported by the evidence currently in hand. For a simple text request:

known_cost = known_input_tokens × input_rate + known_output_tokens × output_rate

with both token terms divided by 1,000,000 when rates are quoted per million tokens. Add cache, reasoning, tool, image, audio, or other units only when the provider’s billing model says they apply and you have the relevant measurement.

Then calculate an estimate for the missing part:

estimated_cost = known_cost + estimated_unobserved_tokens × applicable_rate

State the estimate’s assumption. “We assume generation stopped at the last received chunk” is different from “we estimate the provider completed the requested maximum.” If you do not have evidence for either assumption, keep the status unknown or publish a range rather than a single confident number.

Worked cost example

Assume a hypothetical text model whose pricing snapshot is $2.00 per 1 million input tokens and $8.00 per 1 million output tokens. These are illustrative rates, not a vendor quote. The request sends 12,000 input tokens. The application receives text that its tokenizer counts as 2,400 output tokens before a user cancellation. It has no final provider usage event, so it estimates another 600 output tokens may have been generated before upstream cancellation. No cache, tool, image, audio, or reasoning meter is included in this simplified example.

The lower bound is $0.0432. The provisional estimate is $0.0480. If the provider later reports 2,870 output tokens, the reconciled calculation is $0.04696, before any additional meter. The $0.00104 difference is not automatically an error: it reflects the original estimation assumption. Store it as variance and explain the evidence change.

Do not call the $2.00 and $8.00 values “the provider’s current rates.” A production ledger should load the rate card that applied to the request’s model, region, service tier, cache state, and request date. The AI API invoice reconciliation calculator is the appropriate next step when you have both local attempts and a provider export. Use the model pricing comparison to normalize the rate inputs, and use the AI token cost calculator when the token categories are known but the arithmetic needs checking.

Reconcile instead of hiding the gap

Run reconciliation after a delay appropriate to the provider and endpoint. First match the provider request or response ID. Then compare the provider’s token categories with the ledger’s categories, align the time zone and billing window, and check whether an SDK, gateway, or application retry created another upstream attempt. A request that ended in a client timeout may have a second charge from an automatic retry even when the user saw only one failure.

Keep three numbers visible: the local known lower bound, the provisional estimate, and the provider-reported actual. A dashboard can show estimated spend in a separate color or column without mixing it into an “actual invoice” total. For customer chargeback, choose a policy in advance: pass through provider actuals when available, charge a stated provisional amount and true it up later, or absorb unknowns as an operational loss. The right policy depends on your contract; it is not determined by the stream protocol.

If no provider evidence appears after the retention window, preserve the unknown state and document the exclusion. Do not backfill a zero simply to make totals add up. A zero should mean that your evidence and policy deliberately assign zero, not that the final usage message was lost.

FAQ

Are cancelled streaming API requests free?

Not as a general rule. Cancellation can stop upstream work early, fail to stop it, or occur after billable work has already been generated. Provider and endpoint behavior differ, and a missing final usage event may only mean that the local client did not receive the final metadata. Record the request as exact, estimated, unknown, or reconciled after checking provider evidence.

Should I bill only the tokens displayed to the user?

Not automatically. Displayed text is an application observation, not a universal billing meter. Tokenization can differ from character or word counts, and a request can include input, cached, reasoning, tool, or other billable categories. If your customer contract uses displayed output as its own product meter, that can be a valid product policy, but label it separately from provider cost.

How should a gateway estimate output after a disconnect?

Use a documented estimator and mark the output as estimated. A tokenizer applied to the accumulated text is usually more defensible than a character-only rule, but it still cannot prove tokens generated after the disconnect or account for hidden billable fields. Store the estimator version, tokenization assumptions, price snapshot, and confidence or range. Replace the value when authoritative usage arrives.

What if the provider returns usage several seconds after cancellation?

Keep the record open for a defined reconciliation window and poll or consume the provider’s delayed activity record when the API supports it. Write the late result as a new evidence event linked to the original request. Update the current status to reconciled, but retain the earlier estimate, timestamp, and variance so an audit can explain the change.

Does a retry after a partial response create a second charge?

It can. A retry is a new upstream attempt unless the provider or gateway explicitly guarantees idempotent reuse of the original work. Give the retry its own attempt ID, preserve the first stream’s termination reason, and reconcile both IDs. Never deduplicate solely because the prompts are identical.

What should I show in a finance export?

Include request and attempt IDs, provider and resolved model, pricing snapshot date, input and output fields with evidence states, cancellation and terminal timestamps, status, estimated and actual cost, reconciliation timestamp, and variance. Include a redacted evidence reference, not credentials or sensitive prompt content. This lets finance distinguish a real charge from an estimate and an unresolved unknown.