Research-backed calculator guide

Document Extraction Model Benchmark: Accuracy vs p95 Latency vs Cost per Page

A reproducible way to compare document extraction models on field accuracy, end-to-end p95 latency, and fully loaded cost per page without mixing unlike billing units.

A useful document extraction benchmark does not name a winner from one accuracy percentage. It measures the same corpus and output contract three ways: field-level correctness, end-to-end p95 latency, and fully loaded cost per page. The result should show which system meets your quality and service-level requirements at the lowest cost, not merely which vendor publishes the lowest OCR rate.

What the benchmark must hold constant

Start with a representative, versioned test set. If you process invoices, stratify the sample by supplier, scan quality, language, page count, table density, handwritten content, and layout variation. Do not benchmark only clean PDFs when production includes phone photographs. Keep a held-out test set that is not used to tune prompts, field definitions, preprocessing, or confidence thresholds.

Freeze the output contract. Write down every field, its type, whether it is required, what a missing value means, and the normalization rule. For example, an invoice date may accept 2026-11-02 and 11/02/2026 as the same value after date normalization. An invoice total should usually be compared as a decimal amount with currency, not as an unconstrained string. A line-item table needs both cell values and row/column association; text-only correctness can conceal a layout failure.

Run each candidate on the same files. Pin model version, API region, image resolution, PDF rendering settings, prompt, schema, retry policy, and concurrency. Record request IDs, timestamps, status codes, page count, model configuration, token or page usage, and whether a human corrected the result. If a provider has asynchronous batch processing, measure the actual completion path you intend to operate, not a synchronous toy request.

Accuracy: score fields before averages

A single “accuracy” number is too easy to misread. For each field instance, classify the output as true positive, false positive, false negative, or, where relevant, true negative. Then calculate:

• Precision = TP / (TP + FP), the share of returned values that are correct.
• Recall = TP / (TP + FN), the share of required values that were found.
• F1 = 2 × precision × recall / (precision + recall), useful when both missed and incorrect values matter.
• Document-pass rate = documents in which every required field meets its rule / total documents.

Use exact match for identifiers, dates, tax numbers, and monetary totals when a near miss is not operationally safe. Use a declared partial-match rule for names, addresses, and descriptions. Do not silently use fuzzy matching because it can turn a plausible but wrong vendor name into a “correct” extraction.

Report accuracy by field and document stratum. An overall 97% field score may hide 84% recall on line items or a high-risk failure on tax totals. A useful benchmark therefore includes a small error matrix: field, document type, expected value, returned value, match rule, and failure category. Also track abstentions. A system that returns “needs review” may be safer than one that confidently invents a value, but the abstention still has a labor cost.

p95 latency: define the boundary first

Latency is not one universal number. Decide whether your SLA starts when the file reaches your queue or when your application calls the provider. For an interactive workflow, end-to-end latency should normally include upload, provider queue, model processing, response transfer, parsing, schema validation, and any retry. Keep each component in the log.

Use enough repeated requests to expose tail behavior. A p95 calculated from ten documents is fragile; a larger stratified run gives a more useful view. Sort the complete latency observations and report the percentile method you use. Also report p50, p95, error rate, and timeout rate. If you batch work overnight, report job completion time and throughput instead of presenting batch processing as interactive latency.

Do not compare an online single-page call with a multi-page asynchronous job as if they were the same service. Google’s Document AI limits distinguish online and batch limits; for example, its limits page lists different maximum pages for synchronous and asynchronous processing, and notes that batch jobs are asynchronous rather than designed for real-time applications. ^[4] A provider’s maximum page count is a constraint to model in the workload, not evidence of speed.

Cost per page: show raw and effective cost

Normalize every charge to the same unit. A simple raw vendor cost is:

raw cost per processed page = provider charge / pages processed

A more decision-useful value is:

effective cost per accepted page = (provider + compute + storage + orchestration + retry + review cost) / accepted pages

Keep assumptions visible. Provider prices may vary by region, volume tier, feature, commitment, currency, or contract. Azure explicitly describes its displayed prices as estimates rather than quotes and says actual pricing can vary with agreement, purchase date, and exchange rate. ^[3] AWS examples on the public Textract page use US West (Oregon), while Google and Azure pages describe tiered page meters. ^{[1] [2] [3]}

A feature price is also not a quality label. Plain OCR, layout extraction, key-value extraction, invoice parsing, and generative extraction produce different outputs. Compare the least expensive configuration that satisfies the same output contract. If one system requires a second table parser or an LLM cleanup pass, include that additional call and its latency.

Worked cost example

Assume a US team tests 10,000 invoice pages in one month. The target schema has invoice number, invoice date, supplier, subtotal, tax, total, and line items. The acceptance rule requires exact normalized totals and dates, valid line-item structure, and no unresolved required field. The team estimates that 4% of pages need human correction. A reviewer works at an assumed $30 per hour and spends two minutes per corrected page. The labor rate is an internal estimate, not a vendor quote.

The headline page rate is $0.03, but the effective accepted-page cost is about $0.073 after review. If a second model reduces review to 1% but costs $0.08 per page, it may be cheaper overall: $800 provider cost plus $100 review cost, divided by 9,900 accepted pages, is about $0.0909 per accepted page. That candidate is not cheaper under these assumptions, even though it produces fewer corrections. If its p95 latency is materially lower and that difference prevents an SLA penalty or improves user completion, the business decision may still favor it. Put that value into a separate benefit model rather than hiding it in the extraction price.

For a provider-rate sanity check, the same 10,000-page workload at Google Enterprise Document OCR’s listed $1.50 per 1,000 pages would be $15 before add-ons. Google’s listed custom extraction rate would be $300 before hosting or review. AWS’s example Detect Document Text rate of $0.0015 per page would also be $15 for 10,000 pages in the stated US West (Oregon) tier. Those are arithmetic illustrations from published meters, not an assertion that the three systems deliver the same fields, accuracy, latency, or support. ^{[1] [2]}

A reproducible run sheet

Use this sequence for an invoice extraction latency benchmark or another document class:

1. Define acceptance. List fields, match rules, required values, abstention handling, and document-pass criteria.
2. Stratify the corpus. Label page count, source, scan quality, language, layout, and known edge cases. Freeze a test-set version.
3. Warm up separately. Discard or report warm-up requests; do not mix cold-start behavior into a steady-state number without labeling it.
4. Run the same workload. Use the same concurrency, payload bytes, resolution, region, and retry budget for every candidate.
5. Capture raw events. Store start, queue, first response, completion, status, billed pages or tokens, and correction events.
6. Score outputs. Calculate field metrics, document-pass rate, abstention rate, and results by stratum.
7. Calculate percentiles. Report p50, p95, and p99 latency, plus failures and timeouts. State whether the percentile is end-to-end.
8. Normalize cost. Separate quoted vendor rates, observed usage, internal estimates, and one-time engineering costs. Divide by processed and accepted pages.
9. Plot the feasible frontier. Remove candidates that fail the quality or p95 requirement, then compare effective cost among the survivors.
10. Re-run on change. A provider model version, prompt, OCR preprocessor, page mix, or price change creates a new benchmark version.

The model pricing comparison can help organize vendor-rate inputs, while the LLM cost calculator is useful when an extraction workflow adds token-priced cleanup or validation. If you self-host OCR or a vision model, include GPU idle time and utilization assumptions using the self-hosted GPU inference cost calculator. For tokenized post-processing, the AI token cost calculator provides a separate input/output estimate that should be joined to the per-page extraction meter rather than substituted for it.

How to decide between candidates

Set gates before looking at the totals. An example policy might require document-pass rate of at least 98%, recall of at least 99% on invoice totals, p95 end-to-end latency below eight seconds for interactive requests, and effective cost below $0.10 per accepted page. The exact thresholds depend on the consequences of a wrong field and the workflow’s response-time promise.

Then label each candidate as feasible, review-heavy, latency-limited, or cost-limited. A plain OCR API may be ideal for searchable text but fail the table-structure gate. A structured extractor may pass the schema but require review on blurry scans. A multimodal model may improve variable-layout extraction while adding tail latency and token cost. These are workload findings, not permanent properties of a model family.

Treat prices as dated evidence. Recheck the provider page before procurement, record the region and tier, and request a written quote when volume or commitment pricing matters. Never turn a vendor’s published price into a promise of your invoice total. A benchmark is valuable because it makes those assumptions visible and repeatable.

FAQ

Is OCR character accuracy enough for a document extraction benchmark?

No. Character recognition can be correct while the system assigns a line-item amount to the wrong row or loses the relationship between a key and its value. Measure text accuracy separately from field accuracy and structure accuracy. For an invoice, a valid total and line-item association matter more than a high character score on a header.

What is a good p95 latency definition for extraction APIs?

Use the full path that your user experiences: upload or queue entry through validated output. Report provider-only processing time as a second metric, not as a replacement. Include retries, queueing, parsing, and schema validation when those steps are required for an accepted response. State whether the run is synchronous or asynchronous and whether cold starts are included.

How many pages should a benchmark contain?

There is no universal number. The set should be large enough to represent production strata and to produce stable tail-latency estimates. Start with a held-out set covering every important document type and failure mode, then expand until adding more documents no longer changes the decision. For rare but expensive errors, deliberately oversample the risk in a diagnostic set and report that it is not volume-weighted.

Should human review be included in cost per page?

Yes when review is part of the operating design. Show raw provider cost separately, then add review labor using an explicit correction rate, minutes per correction, and hourly rate. If the reviewer only handles low-confidence pages, measure that queue instead of applying an average to every page. Also report accepted pages, because a cheap pipeline with many rejected outputs may have a high effective cost.

Can I compare Google, AWS, and Azure page prices directly?

Only as a first-pass meter comparison. Their features, page definitions, tiers, limits, regions, and output contracts differ. Google’s OCR, layout, and custom-extraction meters are distinct; AWS charges different Textract APIs and feature combinations; Azure separates Read, prebuilt, custom, query, and add-on meters. Build the same output contract, run the same corpus, and include any second-stage processing before choosing a winner. ^{[1] [2] [3]}