Research-backed calculator guide

Summarization Model Benchmark: Cost per Faithful Summary, Not ROUGE Alone

A reproducible way to compare summarization models on faithfulness, coverage, latency, and cost per accepted summary instead of relying on ROUGE alone.

What this benchmark should measure

A summarization model is doing two jobs at once. It compresses information, and it makes claims about the original. A benchmark that measures only word overlap checks the first job imperfectly and largely skips the second. A better design starts with a fixed test set that resembles production. Include short and long documents, clean and noisy text, routine cases, edge cases, and documents where an omitted detail changes the decision.

Use a stable prompt. Specify the audience, desired format, maximum length, required facts, and the rule that the model must not infer beyond the supplied source. Do not change the prompt for one model after seeing its first output. If you need model-specific formatting adapters, version them and disclose them. Otherwise, you are comparing prompt engineering as well as models.

For each output, score the following dimensions separately:

• Faithfulness: are the claims supported by the input? Check named entities, numbers, dates, causal statements, negations, and commitments. A practical rubric can score each summary from 1 to 5, then define a pass such as 4 or higher with no critical factual error.
• Coverage: did the summary retain the required decisions, risks, action items, and key facts? Use a checklist derived from the source instead of assuming that a reference summary contains every important point.
• Readability: is the result coherent and usable for the intended reader? This can be a human rating, a calibrated judge, or both.
• Conciseness: did the model stop after communicating the needed content? Track summary tokens or words, but do not treat shortness as quality when it removes a necessary qualification.
• Latency and reliability: record p50 and p95 response time, timeout rate, provider errors, and retry count. A model that is inexpensive only when it succeeds is not inexpensive in production.

AWS’s overview groups summarization evaluation into familiar metrics such as ROUGE, METEOR, and BERTScore, but its discussion also makes clear why no single metric captures semantic quality, fluency, and factual consistency. ^[4] The benchmark should therefore report a small scorecard rather than collapse everything into one unsupported “quality score.”

Build a cost-quality test set

Start with a fixed number of documents, such as 1,000, and divide them into meaningful slices. For example, you might have 600 ordinary support tickets, 250 long transcripts, and 150 high-stakes policy or financial documents. The numbers are planning assumptions, not a universal sample-size rule. Keep a separate holdout set for the final decision so prompt tuning does not quietly overfit the benchmark.

For every candidate model, preserve the raw source, prompt version, output, token counts, model ID, timestamp, price snapshot, latency, evaluator result, and error state. This makes the benchmark auditable. It also lets you answer a question that a leaderboard cannot: did a model fail because it hallucinated, omitted a required fact, timed out, or simply produced a longer answer?

Use two evaluation lanes when practical. The first is a cheap, broad automated lane that runs on all outputs. The second is a human-calibrated lane over a stratified sample, with extra review for low scores, judge disagreement, high-risk documents, and outputs close to the pass threshold. If you use an LLM judge, freeze its model, rubric, temperature or equivalent settings, and prompt. Treat judge spend as part of benchmark cost rather than hiding it in engineering overhead.

A benchmark should publish the threshold before inspecting the final table. For instance, define a faithful summary as one with a faithfulness score of at least 4 out of 5, coverage of at least 4 out of 5, and no critical factual error. That is a policy choice for your workload. A legal team may require a stricter rule; a low-risk internal digest may accept a different trade-off. The point is to make the denominator explicit.

Worked cost example

Assume a batch of 1,000 English summaries. Each request contains 4,000 input tokens and produces 500 output tokens. Assume one automated evaluator call per output with 500 input tokens and 100 output tokens. To keep this example inspectable, ignore cache hits, retries, tools, taxes, and batch discounts. The rates below are vendor list prices observed on 2026-11-02: OpenAI GPT-6 Luna at $0.10 per million input tokens and $0.50 per million output tokens; Anthropic Claude Haiku 5.5 at $0.10 and $0.50 per million tokens; and Google Gemini 3.5 Flash-Lite at $0.30 and $2.50 per million tokens. ^{[1] [2] [3]} These are vendor quotes represented on their pricing pages, while every total in the table is my calculation from the stated workload assumptions.

The OpenAI generation total is $0.65, and its evaluator estimate is $0.10, so the measured batch cost is $0.75. With 920 passing summaries, cost per faithful summary is $0.75 ÷ 920 = $0.000815, or about $0.82 per 1,000 faithful summaries. The Anthropic example is $0.75 ÷ 900 = $0.000833, or about $0.83 per 1,000 passing summaries. The Gemini example is $2.45 + $0.10 = $2.55; divided by 950 passes, that is $0.002684, or about $2.68 per 1,000 faithful summaries.

Those figures do not say which model is best. They show how to keep cost and quality in the same unit. If human review costs $0.30 per rejected summary and the 80 non-passing GPT outputs each require review, add $24.00 to GPT’s batch total. Its revised total becomes $24.75, and its effective cost per accepted summary becomes $24.75 ÷ 920 = $0.0269. That is why a token-only comparison can be misleading when quality failures trigger review or rework.

The calculation also exposes an assumption that should be tested: 4,000 input tokens and 500 output tokens may not describe your real distribution. Measure actual p50 and p95 token counts. If long documents are chunked into five map calls plus one reduce call, multiply the input and output work for each stage. If the same policy prompt is reused, model prompt caching may alter the result. OpenAI documents cached-input rates and batch savings on its pricing documentation, Anthropic documents cache writes and cache hits, and Google documents paid-tier caching and batch pricing. ^{[1] [2] [3]} Do not apply any of those discounts until your implementation actually qualifies.

How to report quality without hiding trade-offs

Publish the raw pass rate beside the cost result. A model can win on cost per faithful summary because it has a lower token price, a higher pass rate, a shorter output, or a cheaper evaluator. Those are different mechanisms. Report them separately so another team can reproduce the conclusion.

A useful result row includes model ID, price snapshot date, input tokens, output tokens, generation cost, evaluator cost, total cost, faithfulness pass rate, coverage pass rate, critical-error count, p50 latency, p95 latency, error rate, and cost per passing summary. Include confidence intervals or at least the number of examples behind close comparisons. A 1-point difference from 100 hand-labeled examples should not be narrated as a decisive quality lead.

ROUGE can still help identify regressions in content overlap. METEOR can add a synonym-aware reference comparison, while semantic or judge-based methods can capture some paraphrases that lexical overlap misses. But a high overlap score can coexist with a wrong number or a reversed negation. For high-stakes use, add claim-level checks and human adjudication. This is especially important when a summary will trigger a case decision, update a customer record, or enter a compliance workflow.

Latency belongs in the same decision, but not in the same score unless the business has defined its value. For an overnight batch, a slower model may be acceptable if it reduces rejection work. For an interactive support surface, p95 latency may be a hard constraint. Plot cost per faithful summary against p95 latency and filter out models that fail the required quality floor. This is a constrained decision, not a universal ranking.

For broader monthly planning, connect this benchmark to the LLM cost calculator, then validate the token assumptions with the AI token cost calculator. If one candidate is self-hosted, include utilization, GPU reservation, idle time, storage, and operations rather than comparing its raw inference price with a hosted API. The Self-hosted GPU inference cost calculator is the relevant companion for that case.

FAQ

Is ROUGE useless for an LLM summarization benchmark?

No. ROUGE is a fast lexical-overlap baseline and can reveal whether a system is retaining words or phrases found in a reference. It becomes insufficient when the task permits multiple valid summaries, paraphrase, or source-grounded claims that are not present in a single reference. Keep ROUGE as one column, then add faithfulness, coverage, critical-error checks, and operational cost. The AWS evaluation guide explains both the practical use and the limitations of ROUGE, METEOR, and BERTScore. ^[4]

What is the right denominator for cost per faithful summary?

Use the number of summaries that pass a declared quality policy, not the number of API calls. For example, if a run costs $75 in generation and evaluation and 9,000 of 10,000 outputs pass faithfulness, coverage, and critical-error checks, the cost per faithful summary is $75 ÷ 9,000 = $0.00833. Keep rejected, errored, and manually escalated outputs visible; do not silently remove them from the denominator.

How many documents should a model benchmark include?

There is no universal count. The set should be large enough to cover production slices and to make close decisions less sensitive to a few examples. Start with a fixed pilot such as 1,000 items, stratify by length and risk, and reserve a holdout set. Increase the sample when two candidates are close, when quality variance is high, or when the failure has a high business cost. Report the actual count and slice composition rather than presenting a sample size as a guarantee.

Should the evaluator cost be included in the comparison?

Yes, when the evaluator is part of the decision workflow. Include automated judge tokens, human review time, re-run prompts, and any special verification call in the total benchmark cost. You can also publish a generation-only view for teams that already have an evaluation system. Keeping both views prevents a low-cost model from looking artificially attractive merely because it creates more review work.

How do long documents change the cost calculation?

A long-document pipeline often has multiple map calls followed by one or more reduce calls. Count input and output tokens at every stage, including duplicated overlap between chunks. Measure quality on the final summary and, when possible, inspect intermediate summaries for dropped or invented facts. A large advertised context window does not remove the need to test coverage, latency, and cost on your own document distribution.

When should a more expensive model win?

It should win when its quality or reliability clears a requirement that a cheaper candidate does not, and the added spend is justified by the value of fewer errors, reviews, or missed actions. State that requirement before looking at the final ranking. A frontier model is not automatically better for every slice, and a small model is not automatically cheaper once rejection and human-review costs are included.