Research-backed calculator guide

Retry and fallback idempotency: reconcile duplicate provider charges safely

A practical LLM gateway retry duplicate charge audit: correlate attempts, test idempotency, recompute token cost, and separate confirmed duplicates from uncertain timeouts.

Why retries create a billing evidence problem

A retry is a reliability decision, not a billing classification. Gateways commonly retry timeouts, rate limits, and transient server errors. OpenAI’s error guidance, for example, distinguishes rate-limit responses and server-side 500 or 503 responses from billing, spend, and quota errors that should not simply be retried; it also recommends respecting Retry-After when present. ^[1] Anthropic’s documentation similarly separates rate limits, server errors, timeouts, and overloads, and explains that an SDK may retry transient failures while a streaming error can arrive after an HTTP 200 response. ^[2]

Those distinctions matter because the client’s observation is incomplete. “No response” does not prove “no provider execution.” A gateway can lose a response after the provider accepted the request. A stream can deliver partial output and then fail. A fallback can succeed after a primary attempt that consumed input tokens. If the ledger stores only the final answer, it hides the attempts that created the cost.

Idempotency helps only when it is enforced at the operation boundary and the key remains tied to the same logical operation. Stripe’s primary API guidance illustrates the general pattern: a client supplies a stable idempotency key, the server recognizes later retries, and the server compares parameters so a reused key cannot silently represent a different operation. Stripe also notes that key retention and behavior around validation or concurrent execution are part of the contract, not assumptions a client should invent. ^[3] An LLM gateway should document its equivalent behavior rather than treating a request ID in a log as proof that an upstream provider deduplicated work.

The ledger to build before checking the invoice

Create one row per provider attempt. Keep the logical interaction separate from the attempt. A minimal schema is:

• interaction_id: stable ID for the user-visible operation.
• attempt_id and attempt_number: unique attempt identity and sequence.
• parent_attempt_id: the attempt that caused the retry or fallback.
• idempotency_key: the key sent to a provider or gateway, if supported.
• provider_request_id: the provider’s response or error identifier.
• provider, endpoint, deployment, and model.
• started_at and ended_at, normalized to one timezone.
• trigger: timeout, 429, 5xx, overload, validation error, circuit open, or manual replay.
• outcome: rejected-before-execution, accepted-complete, accepted-partial, timed-out-unknown, or cancelled.
• input_units, cached_input_units, output_units, and other billable meters.
• rate_version or rate-card effective date used for the estimate.
• invoice_period, export row ID, currency, and reconciliation status.

The outcome field should be evidence-based. A gateway timeout with no provider request ID is not automatically “not executed.” Mark it timed-out-unknown until provider usage, gateway logs, or a support response resolves it. Conversely, a provider request ID with usage units is evidence of execution even if the application eventually returned an error.

Do not use a single trace ID as a substitute for this ledger. A trace can connect services, but reconciliation needs a stable join key across your gateway event, provider response, usage export, and invoice line. Preserve raw response IDs and export row identifiers so a reviewer can reproduce the match.

The duplicate decision rule

Use a decision sequence that a finance reviewer can understand:

1. Group by logical interaction. All attempts for one user-visible operation belong together, including fallback models and alternate provider keys.
2. Order by execution time. Keep the gateway timestamp and provider timestamp when both exist. Clock skew should be recorded, not silently corrected.
3. Check idempotency scope. Confirm whether the same key reached the same deduplication layer. A gateway-level key does not prove that two different providers deduplicated the request.
4. Match provider evidence. Join request IDs, usage rows, model, token meters, and invoice period. A matching row is stronger evidence than a client timeout alone.
5. Classify the pair. Mark the later attempt as a confirmed duplicate only when the business operation and billable execution are demonstrably the same. Mark it valid recovery when the earlier attempt was rejected before execution or the later attempt changed provider/model as an intentional fallback. Mark it uncertain when the first execution cannot be established.
6. Escalate with a compact evidence packet. Include both attempt IDs, keys, timestamps, status, usage rows, estimated cost, and the requested provider action. Ask for correction or credit; do not subtract it from actual spend before approval.

A useful tolerance rule is to flag an interaction when a supposedly single operation has more than one completed attempt, repeated keys with mismatched parameters, or token usage on an attempt recorded by the application as “failed before send.” The flag is a review trigger, not a conclusion.

The existing AI API invoice reconciliation calculator is useful after the attempt ledger is assembled. It compares estimated and actual spend, so it can quantify the variance without pretending that a provider invoice can be reconstructed from one blended token rate. For broader planning, use the LLM cost calculator; for unit-level input and output assumptions, use the AI token cost calculator.

Worked cost example

Assume a US-dollar internal estimate. These rates are illustrative assumptions, not vendor quotes: Provider A input is $2.50 per 1 million tokens and output is $10.00 per 1 million tokens; Provider B input is $0.80 and output is $4.00. The example excludes taxes, credits, tool fees, storage, and currency conversion.

The workload has 10,000 logical interactions. The gateway sends 10,000 first attempts to Provider A, then 1,200 retries to Provider A after transient failures, and 600 intentional fallbacks to Provider B. Of the 1,200 retries, the evidence review confirms 450 repeated executions of the same operation. The rest are either valid recovery attempts or still uncertain.

The first-attempt calculation is (15,000,000 / 1,000,000 × $2.50) + (3,000,000 / 1,000,000 × $10.00) = $67.50. The retry slice is $8.10, and the fallback slice is $1.44, producing $77.04 before excluded charges. For 450 confirmed duplicate executions, the assumed per-attempt cost is (1,500 / 1,000,000 × $2.50) + (300 / 1,000,000 × $10.00) = $0.00675; multiplied by 450, the duplicate candidate is $3.0375, rounded to $3.04.

That $3.04 is not automatically a credit. It is the amount supported by the stated assumptions if the 450 rows truly represent duplicate execution. The 750 remaining Provider A retries are $5.06 of potentially valid or unresolved work under the same average-unit assumption. Keep them separate. Calling all $8.10 “double spend” would overstate the evidence.

For a production audit, replace the averages with row-level usage. Preserve cached-input units and any provider-specific meters instead of folding them into input tokens. Then compare the recomputed $77.04 to the provider export and invoice after credits, refunds, taxes, and other charges. The model pricing comparison can help document rate assumptions, but the provider’s dated export remains the billing evidence.

Controls that reduce future duplicate spend

First, put retry ownership in one layer. If both the application and gateway retry, record both budgets and cap their product. A client limit of two attempts combined with a gateway limit of three can create more than three provider calls unless the gateway receives a remaining-budget signal.

Second, generate the idempotency key from the logical operation, not from a new random value on every retry. Include a versioned operation name and a safe hash of material parameters. Never put secrets or personal data into the key. Enforce parameter equality and define key expiry. If the provider does not offer idempotency for a model call, retain the key in your own ledger and classify the result honestly as correlation, not provider-side deduplication.

Third, separate read-only generation from side-effecting tools. A repeated text completion may be wasteful; a repeated “create order,” “send email,” or “charge card” tool call can create a business duplicate. Put the idempotency guard at the tool or transaction boundary as well as at the LLM gateway.

Fourth, alert on unit metrics. Track retries per logical interaction, fallback rate, completed attempts per interaction, unknown-timeout rate, duplicate-candidate dollars, and cost per successful outcome. A rising retry count with flat logical volume is an earlier signal than the monthly invoice.

Finally, retain enough evidence to reproduce a dispute. Keep raw provider request IDs, gateway decision logs, usage export row IDs, the effective rate card, and the reviewer’s classification. Redact payloads where required, but do not discard the metadata needed to prove sequence and execution.

FAQ

Does every LLM retry create a duplicate charge?

No. A retry is an additional attempt, not proof of duplicate execution. The first request may have been rejected before reaching the provider, or the retry may be an intentional fallback after a valid failure. Treat a retry as a duplicate only when the logical operation and billable execution match evidence from the gateway and provider records.

What should I do when a provider times out after the request was sent?

Record timed-out-unknown, preserve the idempotency key and timestamps, and check provider usage or request-ID evidence. Do not mark it free and do not subtract it from the invoice before the provider resolves the case. If a safe idempotency mechanism exists, reuse the same key; otherwise cap the retry and route the uncertainty to review.

Is a gateway request ID the same as a provider request ID?

No. A gateway ID identifies the gateway event. A provider request ID identifies the upstream request and may be required for support or usage matching. Store both, plus the parent attempt relationship. One logical interaction can have several provider request IDs.

How should fallback cost appear in the audit?

Keep fallback attempts as their own provider and model rows under the same interaction ID. A fallback is valid recovery work when the routing policy intentionally tried another provider. It can still be expensive, so include its units and rate in the actual total while keeping it out of the confirmed duplicate bucket unless evidence shows repeated execution of the same operation.

Can I estimate the amount to dispute from token counts alone?

Token counts are necessary but not sufficient. You also need the correct model, rate version, currency, billing period, cached or other meters, and evidence that the rows represent duplicate execution. Token-only estimates are useful for triage; a dispute packet should contain the matched provider usage and invoice evidence.

Which calculator should I use first?

Use the AI API invoice reconciliation calculator when you have an estimate and actual provider data. Use the AI token cost calculator to normalize input and output units, and the LLM cost calculator when you are modeling expected workload cost before an invoice exists.

Limitations and exclusions

This method does not promise that a provider will issue a credit, and it does not infer billing policy from a gateway log. It excludes taxes, refunds, credits, committed-use adjustments, tool charges, embeddings, audio, image meters, and currency conversion unless those rows are added explicitly. Vendor SDK defaults and error classes can change, so re-check the cited documentation and your deployed SDK version on the audit date. The example rates are estimates for demonstrating the calculation, not current vendor pricing.