Research-backed calculator guide

LLM Extraction Benchmark: Field Accuracy, Full-Record Accuracy, Cost and Latency

A practical LLM extraction benchmark method that compares field accuracy, full-record accuracy, cost per accepted record, and latency without hiding normalization or retry assumptions.

What the live results cover

The exact query researched on 2026-11-02 was “LLM extraction benchmark cost accuracy” in English, United States. I also checked the close variants “LLM extraction benchmark field accuracy cost latency” and “document extraction accuracy cost comparison LLM.” The search method was a fresh Manus web search followed by direct HTTPS page fetches. The interface did not reliably expose a country-specific engine setting or ad labels, so the benchmark records that limitation; no returned item was marked sponsored.

The ten returned results use several formats. Research papers emphasize datasets, schemas, exact-match or F1-style scoring, and document-level validity. Vendor benchmark articles show task-specific tables and headline accuracy. Practitioner articles compare extraction cost with OCR or vision workflows. Technical inference guides focus on TTFT, throughput, concurrency, and the latency-cost relationship. A community discussion asks whether general benchmarks predict extraction quality at all.

The recurring coverage is useful but not directly comparable. One result may score every field independently; another may count a document as correct only when every field matches. Some normalize punctuation or permit a numeric tolerance. Others include schema-valid JSON as a success condition, even when a value is semantically wrong. Cost may mean model tokens only, while a production pipeline also pays for OCR, storage, retrieval, retries, and review.

That leaves a specific coverage opportunity: a provider-neutral extraction scorecard that puts field accuracy, all-fields-correct record accuracy, cost per accepted record, and p95 latency in the same worksheet. The goal is not to declare a universal winner. It is to make the assumptions visible enough that a finance, operations, or engineering reviewer can reproduce the comparison.

The four metrics that belong in one benchmark

1. Field accuracy

Field accuracy is the number of correctly extracted fields divided by the total number of evaluated fields:

field accuracy = correct fields / evaluated fields

If 1,000 documents each contain eight required fields, the denominator is 8,000. A field should be marked correct using a written normalization policy. For example, you might treat “$1,250.00” and “1250” as equivalent for a numeric currency field, but not treat “2026-01-02” and “2026-02-01” as equivalent for a date.

Field accuracy is diagnostic. It tells you which fields fail and whether the model struggles with dates, totals, names, long text, or nested arrays. It is not enough as a go/no-go metric when every required value must be usable.

2. Full-record accuracy

Full-record accuracy, also called strict record accuracy, counts a document as correct only when every required field passes validation:

full-record accuracy = fully correct records / evaluated records

If a record has eight fields and one is wrong, the record fails the strict test. This metric exposes error compounding. With independent field accuracy of 97% across eight fields, a rough idealized estimate of all-fields-correct performance is 0.97^8, or about 78.4%. Real fields are not independent, so measure the record result directly instead of relying on that estimate.

You should also report partial acceptance if your workflow allows it. For example, an invoice may be accepted when vendor, invoice number, date, and total are right, while a low-confidence purchase-order description goes to review. That is a different business rule from strict record accuracy and should have a separate label.

3. Cost per accepted extraction

Token price is only an input to the cost model. For a token-billed API, a simple per-document estimate is:

model cost = (input tokens × input price / 1,000,000)
           + (output tokens × output price / 1,000,000)

Then add non-model costs that apply to the same unit: OCR pages, image conversion, retrieval, storage, orchestration, and expected retry cost. Finally divide by the acceptance rate:

cost per accepted extraction = total cost per attempted document / acceptance rate

Use strict full-record accuracy as the acceptance rate when every field is mandatory. Use a business acceptance rate when the workflow has explicit partial acceptance and human review rules. Do not silently substitute field accuracy for acceptance.

Official pricing pages use different model names, tiers, cache rules, and units. OpenAI lists input, output, and cached-input prices per million tokens, while Anthropic lists base input, cache operations, and output per million tokens. Google likewise separates input, output, caching, and feature charges. Check the relevant primary page on the date of your run rather than copying a price from an old comparison. ^{[1] [2] [3]}

4. Latency

Record at least median and p95 end-to-end latency. End-to-end time should include the boundary that matters to your product: upload or retrieval, OCR if used, model request, validation, and retry. If you only measure the model call, label it as model-call latency.

For self-hosted or high-volume inference, also record time to first token, intertoken latency, throughput, concurrency, and queueing. NVIDIA distinguishes TTFT from end-to-end request latency and explains why concurrency and prompt length affect results. ^[7] A short average response at concurrency one does not prove that the same pipeline meets a p95 service target at production load.

Worked cost example

Assume a pilot with 1,000 English-language documents. Each document has eight required fields. The benchmark uses exact semantic comparison after agreed normalization, and a record is accepted only when all eight fields pass. The token rate below is an illustrative use of the OpenAI pricing page’s gpt-6-luna standard rates checked on 2026-11-02: $0.10 per million input tokens and $0.50 per million output tokens. It is a vendor-listed rate, not a forecast of your invoice. ^[1]

The arithmetic shows why “97% accurate” is incomplete. The illustrative field score sounds strong, yet only 74% of complete records pass. The model-only cost per attempted document is one tenth of one cent under the stated rate and token mix, but the retry-adjusted cost per accepted record is higher because 260 records fail strict acceptance and the model is called again for an estimated 8% of attempts.

The retry assumption is deliberately modest and hypothetical. In a real benchmark, measure the retry rate by failure category. A malformed JSON response may be cheap to retry. A wrong extracted value may require a second model, a different prompt, a vision pass, or a human reviewer. Those paths have different costs and should not be averaged into one unexplained multiplier.

If your documents include images, add the image or OCR meter. If prompts contain a repeated schema or policy, test cached-input pricing separately. If you use a batch tier, record the batch discount and any effect on latency or freshness. If you use a gateway, include its fee and any extra tokenization or logging. The AI token cost calculator can help isolate input and output token arithmetic, while the LLM cost calculator is useful for extending the estimate to monthly volume.

How to design a reproducible extraction benchmark

Start with a frozen evaluation set. Use documents that reflect production length, language, scan quality, layout, and field distributions. Split the set into meaningful slices such as clean text, tables, handwriting, multi-page contracts, and long documents. Do not let one easy invoice category dominate the score if production includes difficult policies or statements.

Create a gold record for every document. Store the source document, expected schema, normalized expected values, field-level notes, and an adjudication status. A second reviewer should resolve ambiguous labels before model outputs are compared. Public benchmark audits have shown that noisy or inconsistent ground truth can make model differences look larger or smaller than they really are, so the gold set is part of the measurement system, not clerical overhead.

Define normalization before running models. Specify case folding, whitespace, punctuation, date formats, currency units, decimal precision, null handling, list ordering, and whether equivalent identifiers can be reformatted. Keep two scores when useful: strict exact match and a domain-approved tolerance score. For money, a one-cent tolerance may be appropriate in one workflow and unacceptable in another.

Validate structure separately from meaning. A JSON schema can prevent missing required keys, wrong types, or invalid enum values, but a schema-valid value can still be factually wrong. OpenAI describes Structured Outputs as schema adherence with programmatically detectable refusals; Anthropic documents constrained, parseable JSON plus refusal and token-limit cases; Google documents JSON Schema support for extraction. These features reduce formatting failures, not the need for semantic evaluation against a gold record. ^{[4] [5] [6]}

Run the same prompt, schema, temperature or sampling policy, context, and post-processing across candidates. Record model version, endpoint, region, service tier, date, input and output token counts, cache hits, tool calls, retries, and failure reasons. Microsoft’s published evaluation example is useful here because it explicitly fixes prompting, measures execution time, estimates cost from token averages, and runs multiple document variants before comparing accuracy. ^[8]

For latency, warm the system according to a declared protocol and report both warm and cold behavior when relevant. Sweep a small concurrency range rather than testing only one request. Record queue time and timeouts. At minimum, report p50 and p95; add p99 when a missed deadline is expensive. Keep throughput separate from user-perceived latency.

Reading the result without overclaiming

A Pareto-style comparison is more honest than a single winner. A model is attractive when no alternative is both cheaper and better on the metrics that matter. A high-accuracy, high-cost model may be rational for a small set of regulated records. A cheaper model may win for low-risk fields if its accepted-record rate remains above the operational threshold.

Use a two-stage design when the error distribution supports it. A low-cost model can process ordinary documents, while a validator or stronger model handles low confidence, schema failures, or difficult document slices. Compare the complete pipeline against a single model. The extra validation call may increase cost but reduce human review and rework.

Do not convert a benchmark into a traffic or savings promise. A test set is evidence about the tested task under stated conditions. It does not establish search demand, future model quality, indexing, rankings, revenue, or a guaranteed production outcome. Re-run the benchmark when the model, tokenizer, prompt, document mix, schema, or price changes.

For a broader monthly model comparison, use the model pricing comparison and document the assumptions beside the output. If you are considering self-hosting, the self-hosted GPU inference cost calculator can extend the same logic to utilization, hardware, and throughput. Keep the unit consistent: cost per attempted document, cost per accepted document, and cost per reviewed exception are different measures.

FAQ

Is field accuracy the same as extraction accuracy?

No. Field accuracy averages individual values, so a record with seven correct fields and one wrong field can still contribute 87.5% field accuracy. Strict full-record accuracy marks that same record as failed. Report both when a downstream system needs complete records.

What is the best denominator for cost per accepted extraction?

Use the number of records that pass your declared acceptance rule. If all fields are mandatory, divide total attempted cost by fully correct records. If some fields can go to review, define the partial-acceptance rule and report human-review cost separately rather than hiding it in the model cost.

How many documents are enough for a benchmark?

There is no universal count. The set should be large enough to represent important document and field slices, and it should include enough examples of rare failure modes to make the result useful. Report the sample size per slice, not only the combined total. Add confidence intervals or repeated runs when a small score difference could change the decision.

Should schema-valid JSON count as a correct extraction?

Only for a schema-validity metric. Valid JSON proves that the output is parseable and structurally compatible. It does not prove that the invoice total, date, or name matches the source. Score syntax, schema adherence, field semantics, and full-record correctness separately.

How should latency be measured for a document extraction API?

State the start and stop boundaries. For an end-to-end workflow, include upload or retrieval, OCR, model time, validation, and retries. Report p50 and p95 under a stated concurrency and document-size mix. If you report only time to first token or only the model call, label it clearly; those are not interchangeable with completed-record latency.

Do vendor pricing pages give a final cost forecast?

No. They give rate-card inputs. Your forecast still needs measured input and output tokens, cache behavior, batch or priority tier, image or OCR charges, tool calls, retries, storage, and review. Date the rate and preserve the calculation so it can be rerun when pricing or model versions change.