Research-backed calculator guide

Coding Model Benchmark: Cost per Passing Task, Context Reuse and Harness Effects

A practical method for measuring coding-agent quality as verified passes per dollar, with cache reuse, harness, retry, and evaluator effects made explicit.

What “quality” should mean in a coding benchmark

Start with an executable acceptance rule. For a repository task, a binary pass can mean that the patch applies, the new regression test changes from failing to passing, and the existing test suite still passes. That is close to the construction and evaluation logic described in the SWE-bench paper.^[5] It is also consistent with the official SWE-bench leaderboard’s use of “% Resolved” as the percentage of task instances solved.^[3]

A binary result is useful because it prevents a low token bill from being mistaken for useful software. However, it is incomplete for production decisions. Keep these additional fields in the run ledger:

• task_pass: whether the required checks passed.
• regression_free: whether unrelated tests stayed green.
• review_score: a human or independent judge score for maintainability, scope, and security.
• turns, input_tokens, cached_input_tokens, and output_tokens.
• wall_time, tool errors, retries, and the reason a run stopped.

Use pass rate for the headline quality metric only when every candidate sees the same task set and evaluator. Otherwise, call the number a result for that particular benchmark configuration. SWE-Bench Pro highlights the problem with casual comparisons: it uses diverse long-horizon tasks and reports a primary resolve rate, while the official SWE-bench site presents several variants, including Lite, Verified, Multilingual, and Multimodal.^{[2] [3]}

The cost formula

For a managed API, calculate each task’s token bill from the provider’s meter and then add any separately billed tool calls. A simple form is:

run_cost = input_tokens × input_rate + cached_input_tokens × cache_rate + output_tokens × output_rate + tool_costs

Use rates in dollars per million tokens by dividing token counts by 1,000,000. Do not count cached tokens again at the ordinary input rate. If the provider reports cache writes, cache reads, reasoning tokens, or tool tokens separately, retain those fields rather than folding them into an uninspectable average.

For a self-hosted model, a more defensible first estimate is:

run_cost = GPU-seconds × measured dollars per GPU-second + allocated fixed costs

Measure throughput on the same context shape and concurrency used by the benchmark. The AWS coding-agent methodology explicitly distinguishes per-token managed billing from GPU-time pricing, and recommends varying model, harness, and hosting as separate levers rather than assuming that one provider price represents the whole frontier.^[1]

Then aggregate:

pass_rate = passed_tasks ÷ attempted_tasks

cost_per_passing_task = total_run_cost ÷ passed_tasks

Do not divide by the number of tasks that received a response. A failed run consumed resources too, and its cost belongs in the numerator. Do not hide retries: either include retry tokens in the task total or report a separate “first attempt only” view.

Worked cost example

The following is an illustrative API calculation, not a vendor quote for a benchmark run. It uses the OpenAI pricing page checked on November 2, 2026, which displayed GPT-6.1 Sol at $2.00 per 1 million ordinary input tokens, $0.10 per 1 million cached input tokens, and $10.00 per 1 million output tokens.^[6] The example assumes a fixed 30-task set, one attempt per task, and no separate tool or hosting charge.

Assumptions: each task consumes 180,000 input tokens, of which 70% are cache hits; each task generates 30,000 output tokens; 27 of 30 tasks pass the evaluator; and all prices are USD.

If the same 180,000 input tokens were all billed as ordinary input, the input portion would be $0.36 per task rather than $0.1206. Total cost would be $19.80, or $0.7333 per passing task. The cache assumption therefore changes the decision by $0.2660 per passing task in this constructed case. It does not prove that caching will produce that saving in your harness: cache eligibility, prefix stability, cache-write fees, eviction, and provider accounting must be measured.

A benchmark can also compare quality at a fixed budget. For example, a 30-task run that passes 27 tasks at $12.618 has a 90% pass rate and $0.4673 per pass. A second configuration that passes 25 tasks at $8.00 has a lower total bill but costs $0.32 per pass. Whether the second is preferable depends on the value of the two additional successful changes, review effort, and failure consequences. The calculation does not declare a winner by itself.

Context reuse is part of the benchmark, not a footnote

Coding agents often resend repository instructions, file excerpts, tool schemas, and conversation state. Two runs using the same model can therefore have different bills if one harness preserves a stable prefix and the other reconstructs it on every turn. Anthropic’s API documentation describes prompt caching as a way to reuse previously processed content, with distinct cache-write and cache-read meters; it also states that cache reads are priced below base input for the documented models.^[7]

Log at least four context fields per request: ordinary input, cache-write input, cache-read input, and output. Also record the cache key or prefix version in a privacy-safe way. A benchmark that reports only “tokens per task” can hide whether the result came from a cold context or a warm repeated workflow. For longer coding tasks, the distinction can dominate a model-price comparison.

Google’s Gemini pricing documentation similarly separates input, output, context caching, and batch or tool pricing, and notes that thinking tokens can be included in output billing for the listed models.^[8] That means the run ledger must match the provider’s billing semantics. Never infer billed tokens from visible text alone when the provider exposes reasoning or tool meters.

Harness effects can overwhelm a model-only comparison

A harness controls prompt construction, file access, tool calls, subagents, recovery, permissions, and stopping rules. Harness-Bench describes the agent as a model plus harness and evaluates configurations under shared task, budget, timeout, and evaluator conditions; it records final artifacts, traces, usage statistics, and validator outputs.^[4] Its framing is a practical warning: “the model” is not the whole system that produced a patch.

A coding-agent cost comparison should therefore hold the harness fixed when comparing models. Then run a second experiment that holds the model fixed while comparing harnesses. Report the number of tool calls, subagent invocations, retries, and total context tokens. The AWS methodology gives a concrete example of why: a heavier harness in its measured runs increased tokens, cost, and wall-clock time without a matching quality gain on the tested tasks.^[1] Treat those figures as that study’s observations, not a guarantee for your codebase.

For a fair harness experiment, pin the repository commit, task prompt, timeout, model endpoint, temperature or sampling settings, test command, and evaluator. Keep the maximum budget visible. A harness that is allowed twice as many turns may achieve a higher pass rate, but its cost-quality frontier is not directly comparable to a capped run.

How to report SWE-bench cost per solved task

If you use SWE-bench, name the exact variant and runner. The official site lists 2,294 instances for the original benchmark, 500 for Verified, and 300 for Lite, alongside other variants.^[3] The original paper describes tasks derived from merged pull requests and evaluates a generated patch by applying it and running the task tests; a task is resolved when those tests pass.^[5] Use those definitions instead of treating a leaderboard percentage as a general software-engineering success rate.

For every submission, capture the benchmark’s resolved count, total attempted count, provider token usage, harness version, per-task cost, capped or uncapped budget, and any excluded or timed-out tasks. Then show both:

SWE-bench cost per solved task = benchmark run cost ÷ resolved task count

and:

SWE-bench cost per attempt = benchmark run cost ÷ attempted task count

The first is useful for unit economics. The second makes it harder to overlook expensive failures. If a leaderboard or paper does not expose token or infrastructure cost, do not backfill it from a model’s public list price and call it measured cost. Label that calculation as an estimate and state the token, cache, retry, and tool assumptions.

Scale’s SWE-Bench Pro page illustrates another boundary: it reports a distinct task pool and strict resolve-rate conditions covering issue resolution and no regressions.^[2] A result from Pro should not be merged into an average with Verified unless the task design, evaluator, and cost accounting are intentionally normalized.

A practical benchmark worksheet

Create one row per task and one summary row per model-harness-hosting configuration. Minimum task columns are task_id, repo_commit, model, harness_version, attempt, passed, regression_free, input_tokens, cache_write_tokens, cache_read_tokens, output_tokens, tool_cost, gpu_seconds, wall_seconds, and failure_reason. Keep raw provider usage records so an invoice reconciliation can reproduce the summary.

Before comparing configurations, pre-register exclusions. Decide whether a provider outage, test flake, malformed task, timeout, or policy refusal is a failed attempt, a rerun, or an excluded observation. Apply the same rule to every candidate. If a second attempt is permitted, report both first-attempt pass rate and eventual pass rate, along with the incremental cost of recovery.

Use confidence intervals or at least task-level counts when differences are small. A one-pass difference in a small task set is not the same evidence as a stable gap across hundreds of tasks. Stratify by repository, language, change size, and issue type where the task set supports it. A model may look cheaper overall because it receives more easy tasks, while another handles the hard slice that matters operationally.

Finally, refresh the benchmark when the model, price card, cache policy, harness, evaluator, or repository changes. Provider pricing pages are time-sensitive. OpenAI documents separate standard, cached, batch, and service-tier choices; Anthropic documents cache operations; Google documents tier and model-specific rates.^{[6] [7] [8]} Record the checked date and unit for every price. A benchmark without a date is not a durable cost baseline.

FAQ

Is cost per passing task better than cost per million tokens?

For a coding decision, usually yes. Tokens are an input to the run, while a passing task is the outcome you are paying for. Keep cost per million tokens as an audit field, but use total run cost divided by verified passes to connect spend with quality. The ratio is meaningful only when the task set and evaluator are held constant.

Should a failed coding task count in the denominator?

No. Its spend belongs in total cost, but it is not a passing task. If 30 attempts cost $12 and 24 pass, the cost per passing task is $0.50, not $0.40. Also report the 80% pass rate so readers can see both the numerator and denominator.

How should cached context be priced?

Use the provider’s actual cache-read and cache-write meters. Separate cache hits from ordinary input in the ledger. If you only have a list price and an assumed hit ratio, label the result an estimate and show the assumption, because prefix stability, cache duration, and eviction can change the billed amount.

Does a higher SWE-bench score mean a better production coding agent?

Not automatically. SWE-bench measures resolution under a specified dataset, environment, harness, and test evaluator. The official benchmark variants and newer Pro-style datasets have different scopes and difficulty. Use the score as evidence for that evaluation, then test representative private tasks before making a production selection.

Should the harness be included in model quality reporting?

Yes. Report model and harness as a pair, then run controlled ablations when possible. Tool policy, context compaction, recovery, subagents, and stopping rules can change both pass rate and cost. A model-only label obscures the system that actually produced the patch.

What costs are excluded from the worked example?

The example excludes human review, CI minutes, repository hosting, observability, provider fixed fees, tool-call charges, retries, and self-hosted GPU time. Add those meters when they are material. The example is an API-token illustration, not a complete total-cost-of-ownership quote.

Related calculations

Continue the estimate with LLM cost calculator, compare the model assumptions with AI token cost calculator, and check realized charges using Self-hosted GPU inference cost calculator.