Research-backed calculator guide

Reasoning Model Benchmark: Cost per Correct Answer and Token-Efficiency Frontier

A practical reasoning-model benchmark that measures cost per correct answer, reasoning-token efficiency, retries, and the feasible accuracy-cost frontier.

What to measure in a reasoning benchmark

Start with a frozen evaluation set. It should represent the production job rather than a collection of questions that happen to be convenient. Include enough items in each important slice—such as arithmetic, multi-step analysis, code, knowledge, and tool-use planning—to reveal where reasoning effort changes. Keep a hidden answer key or executable test suite, and avoid changing prompts between models. If a task has multiple acceptable answers, define the rubric before running the benchmark.

For each attempt, store at least these fields:

• model_id and reasoning setting, such as low, medium, high, or a provider-specific budget;
• prompt and input-token count, including any repeated context;
• reasoning-token count when the provider exposes it;
• visible output-token count and total billed output tokens;
• latency and whether the request stopped at a limit;
• correctness, grader confidence, and the reason for a failure;
• retry or escalation status; and
• the price sheet version and currency used for the calculation.

Do not silently compare provider counters as though they mean exactly the same thing. One API may expose a separate reasoning count, another may expose thinking tokens, and another may expose only total completion tokens. The Nous Research report describes this problem directly: closed models may summarize or hide raw reasoning, and reported reasoning counts can differ in reliability. A defensible benchmark therefore uses the most comparable billed completion measure available, labels provider-specific fields, and keeps raw usage records for later reconciliation.

OckBench provides a useful research pattern. Its revised paper uses Pass@1 correctness, counts raw generated output tokens, and applies a differentiation filter to select tasks where model accuracy is neither near zero nor near one and token consumption varies across models.^[1] That filter is valuable because trivial questions can make every model look equally efficient, while impossible questions can make every model look equally wasteful. For a production benchmark, use the same principle: select slices where a purchasing decision could realistically change.

Cost per correct answer beats cost per token

A rate card is only an input to the decision. A lower output rate can be offset by longer reasoning, more failed first attempts, or a larger retry rate. Conversely, a higher-priced model can be cheaper per successful task if it solves more cases in one pass. The AI token cost calculator is useful for checking the input, output, and cached-token arithmetic before adding correctness and retry terms.

Use a two-level calculation. First, calculate the raw cost of each attempt:

attempt_cost = (I / 1,000,000 × P_i) + (R + O) / 1,000,000 × P_o

Here, I is input tokens, R is reasoning or thinking tokens, O is visible output tokens, P_i is the input price per million tokens, and P_o is the output price per million tokens. If the provider gives only total output tokens, use that total instead of R + O; do not add a separately reported reasoning count a second time.

Then calculate the benchmark result:

cost_per_correct = sum(attempt_cost) / correct_first_attempts

This measure intentionally treats an incorrect first answer as spend that did not purchase a correct completion. If your production system retries failures, model the retry policy explicitly. For example, if 100 tasks produce 82 correct first attempts, 12 corrected answers after one retry, and 6 unresolved failures, divide total spend by 94 successful outcomes only if “successful outcome” is your business definition. If the decision is specifically first-attempt reliability, divide by 82 instead. Both can be useful, but they answer different questions.

OpenAI’s documentation warns that a response can become incomplete when generated tokens reach max_output_tokens, even before visible output appears, and that input and reasoning tokens can still incur cost.^[2] Record these incomplete responses as failures or censored attempts according to a predeclared rule. Do not remove them because they make a model look inefficient.

Worked cost example

Assume three model settings are tested on the same 1,000-question reasoning set. The numbers below are illustrative assumptions, not vendor quotes. Prices are shown in USD per million tokens and held constant only to make the arithmetic inspectable. The benchmark date is 2026-11-02. Each setting uses the same $2.00 input rate and $12.00 output rate for this example; substitute the applicable dated rate card before making a purchasing decision.

For setting A, output tokens equal 800, so output cost is 800 / 1,000,000 × $12 = $0.0096. Total benchmark spend is $0.0136 × 1,000 = $13.60, and 860 answers are correct on the first attempt. Its cost per correct first attempt is $13.60 ÷ 860 = $0.0158.

Setting B raises first-attempt accuracy by six percentage points but almost doubles cost per correct answer. Setting C adds two more accuracy points but costs about 3.5 times setting A per correct first attempt. If the business requirement is at least 92% first-attempt accuracy, B is the least-cost feasible choice in this small example. If 94% is mandatory, C is feasible, but the benchmark should still test whether a fallback strategy can reach that reliability at lower expected cost.

The table also shows why “reasoning tokens versus accuracy” should be plotted as a curve rather than summarized by one winner. Increasing effort can improve accuracy with diminishing returns. If a low setting is cheaper but misses a contractual quality threshold, it is not a feasible option. If high effort improves accuracy only on one slice while wasting tokens on easy tasks, a routed policy may dominate every fixed setting.

Build the token-efficiency frontier

For each model setting, plot average billed output tokens on the x-axis and accuracy on the y-axis. Mark settings that are dominated: a point is dominated if another point has equal or higher accuracy and fewer tokens. Repeat the plot using cost per attempt, because two providers can have similar token counts but different input/output prices.

Do not force one scalar score too early. A composite score can hide a policy choice about how much accuracy is worth. If you need one, publish the formula and sensitivity-test it. OckBench’s paper treats accuracy as the base term and token use as a logarithmic penalty, while its project page presents an OckScore that combines accuracy with average-token usage.^[1] Those are research design choices, not a replacement for your workload’s service-level objective.

A stronger production decision uses three gates:

1. Quality gate: minimum first-attempt accuracy for every critical slice, not just the pooled average.
2. Cost gate: maximum cost per correct answer, including expected retries and escalation.
3. Operational gate: maximum p95 latency, incomplete-response rate, and context-limit failure rate.

Then choose the cheapest setting that passes all three gates, or route by slice. A single pooled average can conceal a model that is excellent on easy knowledge items and unacceptable on code or safety-critical reasoning.

Provider settings change what “effort” means

Treat a label such as “high” as a control, not as a universal unit of thinking. OpenAI says reasoning.effort is model-dependent and can include values such as low, medium, high, and xhigh; lower effort generally favors speed and lower token usage, while higher effort asks for more complete thinking.^[2] The same page says reasoning tokens occupy context and are billed as output tokens.

Gemini’s official guidance exposes thinking_level controls and reports total_thought_tokens separately from total output tokens.^[4] It also warns that a hard max_output_tokens cutoff includes thought tokens and can yield an incomplete or empty response while still billing generated thinking. That means “cap the output” is not equivalent to “make reasoning efficient.” Test both effort control and failure behavior.

Anthropic’s manual extended-thinking documentation says budget_tokens is a target, actual use varies, and the budget must leave room within max_tokens for the final response.^[5] It also describes migration toward adaptive thinking on newer models. Freeze the API configuration and model snapshot in the benchmark record; otherwise, a later rerun may be comparing different control semantics under the same friendly label.

How to run the benchmark responsibly

Run at least two passes when sampling noise matters: a deterministic or low-temperature pass for repeatable regression checks, and a production-like pass that reflects your deployed sampling settings. Use a fixed random seed where supported, but do not assume a seed makes hosted model behavior permanently reproducible. Save prompts, model IDs, settings, usage responses, grader outputs, and timestamps.

Report confidence intervals or bootstrap intervals for accuracy and cost per correct answer when the dataset is not large. A two-point accuracy difference on 100 tasks may be noise; the same difference on 10,000 representative tasks may be operationally meaningful. Break out easy, medium, and hard slices. Also report the distribution of reasoning tokens: p50 and p95 often matter more to a budget owner than the mean.

Separate model capability from benchmark harness effects. A shorter output may reflect an answer-format constraint, hidden truncation, or a grader that accepts an unsupported guess. Check that every accepted answer meets the same correctness rubric. For code, run tests rather than judging prose. For numeric answers, normalize formatting but not substance. For open-ended answers, use blinded human or calibrated model grading and audit disagreements. If the benchmark will compare hosted APIs with self-hosted deployments, the self-hosted GPU inference cost calculator can provide a separate infrastructure-cost baseline rather than mixing GPU overhead into token prices.

FAQ

Is cost per million tokens enough to choose a reasoning model?

No. It ignores how many input, reasoning, and visible-output tokens the model uses, as well as first-attempt accuracy and retries. Use the rate card to calculate attempt cost, then divide total spend by the number of correct outcomes relevant to your decision.

How do I benchmark hidden reasoning tokens?

Prefer the provider’s billed usage fields when available. If only total completion tokens are exposed, use that total as the comparable measure and label it clearly. Do not pretend a visible answer length reveals internal effort. Store the raw usage object so the calculation can be updated when provider reporting changes.

Should a benchmark reward fewer tokens even when accuracy falls?

Only after applying a quality gate. A cheaper wrong answer is not a successful completion. After minimum quality is met, token use and cost can distinguish efficient settings. If the task has a safety or contractual threshold, optimize inside that feasible region rather than trading away required correctness.

What is a Pareto-efficient reasoning setting?

It is a setting for which no tested alternative is both at least as accurate and no more expensive or token-intensive. Several settings can be efficient at once. The frontier changes when you change the task mix, price assumptions, retry policy, or required accuracy.

How often should I rerun the benchmark?

Rerun it when a model snapshot, reasoning control, price sheet, prompt template, grader, or production task mix changes. For a stable service, schedule a smaller regression suite regularly and run the full golden set at release gates. Keep old results instead of overwriting them so cost and quality drift remain visible.