Research-backed calculator guide

Human Annotation Cost per AI Evaluation Task: Labels, Double Review and Rework

Calculate human annotation cost per AI evaluation task by separating reviewer time, double review, disagreement adjudication, and rework. Includes a transparent cost-per-final-label example.

Human annotation cost per AI evaluation task is not one stable price. It is the fully loaded cost of producing one accepted evaluation label after the work has passed through the review design you actually use. A one-pass binary label, a pairwise preference judgment, and a rubric with senior adjudication are different cost units even when each is called “one task.”

What the cost unit should mean

Define the denominator before collecting rates. “Per label” can mean a worker’s first click, a completed review form, or a final label that has survived quality control. Those denominators are not interchangeable. For budgeting, the most useful denominator is usually one final accepted label: the item is reviewed, disagreements are handled, required corrections are made, and the output is ready for analysis or training.

A final label may contain several judgments. A binary safety label may need two independent reviewers. A helpfulness rubric may contain five scored dimensions. A pairwise preference task may require a comparison, a reason code, and a tie option. Count the review actions that consume time, not merely the number of rows in the export.

NIST’s AI RMF measurement guidance calls for documented test sets, evaluation tools, metrics, human-subject requirements where applicable, and evidence that performance is valid and reliable in conditions similar to deployment. ^[1] That is a useful budgeting principle: the cost model should describe the evaluation design and its controls, not just multiply a row count by an attractive piece rate.

A practical cost model

Use these variables for one task or one homogeneous batch:

• N = number of items evaluated.
• R = independent first-pass reviewers per item.
• T = seconds for one reviewer to complete one first-pass judgment.
• H = loaded hourly cost for that reviewer type. Include pay plus employer costs or vendor charges that are genuinely part of the rate.
• D = share of items sent to adjudication because reviewers disagree or flag uncertainty.
• A = seconds for one adjudication and H_a = loaded hourly cost of the adjudicator.
• Q = share of items requiring rework after quality control.
• W = seconds for one rework review and H_w = loaded hourly cost of the rework reviewer.
• P = per-item platform, recruiting, or management allocation that you intentionally include.

The batch estimate is:

Total cost = N × [(R × T ÷ 3,600 × H) + (D × A ÷ 3,600 × H_a) + (Q × W ÷ 3,600 × H_w) + P].

Divide total cost by the number of final accepted labels to get the unit cost. If rework creates extra final labels or duplicates, use the count of accepted outputs in the denominator rather than assuming it equals the original item count.

Disagreement and rework are separate inputs. Disagreement means the first-pass judgments do not resolve to the rule you specified. Rework means a quality check sends an item back for correction. They may overlap, but they do not have to. If every disagreement automatically triggers rework, measure the joint rate and avoid adding the same labor twice. If a reviewer’s “rework” is actually senior adjudication, put that time in the adjudication term instead.

Do not hide idle time, training, calibration, or queue management inside the seconds-per-label assumption unless you can explain it. You can either use a loaded hourly rate that already includes those costs or add a separate batch-level overhead line. Mixing both methods inflates the result.

Worked cost example

Assume a U.S.-based evaluation pilot with 1,000 text outputs. The figures below are planning assumptions chosen to demonstrate the method. They are not quoted market prices.

The first-pass term is easy to underestimate. Two 45-second judgments consume 90 reviewer-seconds per item, not 45. The disagreement term is also meaningful: 180 items receive an additional 90-second senior review. The rework term covers another 100 items, but it is deliberately modeled separately so the team can replace the 10% assumption with a measured post-QA rate.

Now test the sensitivity. If first-pass review rises from 45 to 75 seconds while every other input stays unchanged, the first-pass subtotal becomes $1,000 and the batch total becomes $1,242.50, or $1.24 per final label. If the disagreement rate falls from 18% to 8%, adjudication falls from $202.50 to $90 and the total becomes $730, or $0.73 per label. In this example, time per judgment and disagreement rate are stronger levers than rounding a reviewer rate by a few cents.

A three-reviewer design is not automatically cheaper because it may reduce unresolved disagreements. Model both effects. For example, three 45-second reviews at $24 per hour cost $0.90 per item before adjudication. If that design reduces disagreement enough to save senior time and rework, it may be justified for a high-stakes slice. The correct comparison is cost per accepted label at the same quality target, not cost per initial click.

How double review, disagreement, and rework change the estimate

Double review. Set R to two when two people independently judge every item. Do not use an average hourly rate that quietly assumes one reviewer. Double review also gives you a measurable disagreement rate, which is often more useful than a vague “quality buffer.” For a stratified program, use separate R and T values for generalist, specialist, and senior reviewers rather than forcing them into one blended rate.

Disagreement. Decide what counts as disagreement. Two ratings may differ on a five-point scale by one point, or only a direct contradiction may trigger adjudication. Write the rule before calculating. The disagreement percentage should come from a pilot or historical batch whenever possible. If the sample is small, show a range, such as 10%, 18%, and 30%, rather than presenting one forecast as a fact.

Rework. Track rework by cause. A guideline defect, an annotator mistake, a tooling failure, and a genuinely ambiguous model output have different remedies. A single rework percentage hides these drivers and makes it hard to improve the process. Record the extra seconds and the role that performs the correction. If an item is fully re-annotated by two reviewers, its rework cost is closer to another first-pass panel than to a quick correction.

Quality controls. Gold items, calibration sessions, audits, and escalation queues consume labor even if they do not appear in the final label file. Add them as explicit batch costs or incorporate them into a defensible loaded rate. NIST also recommends regular assessment, expert or independent involvement where appropriate, and documentation of limitations and generalizability. ^[1] These controls are not proof that a label is correct, but omitting them makes the budget look artificially low.

Choosing a rate without inventing precision

A reviewer rate should state its currency, geography, date, role, and inclusions. This article uses USD assumptions for an English, United States planning example. A vendor invoice, internal salary conversion, or contractor payout is a different evidence type from a public wage statistic. The BLS Quarterly Census of Employment and Wages reports employer-reported employment and wages for more than 95% of U.S. jobs at several geographic and industry levels, but it is not an AI-annotation-specific rate card. ^[2] Use it as context or a source for a broader labor-cost benchmark, not as a made-up “human evaluation price.”

For an internal estimate, convert annual compensation to an hourly loaded rate with a documented productive-hours assumption. For a vendor estimate, use the quoted billable rate and list what it includes. Do not combine a worker payout from one country with a U.S. employer burden and call the result a market average. Localization, language, domain expertise, sensitive-content handling, and procurement minimums can move the real rate materially.

Run a small pilot and measure median and upper-percentile review time. A mean alone can understate a long tail of difficult cases. Keep ordinary items and escalated items separate. If 80% of rows take 30 seconds and 20% take 120 seconds, a single “average” is less actionable than two strata with different reviewers and adjudication rules.

When an LLM judge changes the human budget

An automated judge can reduce the number of human reviews, but it does not make human cost disappear. You still need calibration data, a validation sample, monitoring, escalation rules, and an allowance for cases the judge handles poorly. A 2026 arXiv paper proposes treating LLM ratings as auxiliary measurements while retaining a designed subsample of human ratings, rather than using agreement as a one-time reason to discard human oversight. ^[3]

Translate that idea into the cost model with a routed share. If an LLM screens all N items and sends fraction S to humans, model human first-pass, adjudication, and rework terms on S × N. Add the LLM inference cost, calibration-set review, and ongoing audit cost separately. Do not assume the human rate scales linearly if the routed cases are the hardest cases; their review time and disagreement rate may be higher than the ordinary queue.

Use the AI agent budget guardrail and P95 calculator to pressure-test a routed workflow’s tail budget, and compare its ongoing review spend with the broader AI Agent Cost Calculator. If review events are part of a production monitoring program, the AI observability ROI calculator can help separate evaluation labor from incident and observability benefits. For reconciling the modeled estimate with actual provider or platform charges, see the AI API invoice reconciliation calculator.

FAQ

What is a reasonable human annotation cost per AI evaluation task?

There is no responsible universal number without the task definition. The defensible unit is a documented estimate or quote based on seconds per judgment, reviewer count, loaded rate, disagreement handling, rework, and overhead. A simple one-pass label can be much cheaper than a specialist rubric with double review and adjudication. Report a range when pilot data is uncertain.

How do I calculate the cost of double annotation?

Multiply the time for one judgment by two independent reviewers, divide by 3,600, and multiply by the loaded hourly rate. Then add expected adjudication and rework. If the second reviewer is a specialist or senior reviewer, use that person’s own time and rate rather than doubling a blended average.

Should disagreement be included in cost per label?

Yes, when the denominator is a final accepted label. Multiply the share of items that reach adjudication by adjudication seconds and the adjudicator’s loaded hourly rate. If disagreement is only measured for an audit sample, apply it only to that sample and state the sampling design. Never count the same labor again under rework.

Is rework the same as human review?

Rework is additional labor after an initial review or quality check. It may be a quick correction, a full re-annotation, or a senior adjudication. Model it using the actual correction path. If a failed item receives a complete second panel, its cost can be close to the original review cost.

Can an LLM judge replace human annotators to lower cost?

It can reduce the human-reviewed share for some workflows, but replacement is a validation decision, not a default pricing assumption. Keep a representative human sample, test agreement and construct validity, and allocate more human review to segments where the automated judge is less predictable. The arXiv design discussed above treats human ratings as the primary quantity of interest and LLM ratings as auxiliary evidence. ^[3]

What should a vendor quote include?

Request the currency, region, worker type, task definition, expected seconds per judgment, reviewer multiplicity, quality thresholds, disagreement and adjudication policy, rework responsibility, platform charges, minimum batch, turnaround assumptions, and treatment of sensitive content. Ask whether the price is per first-pass label or per final accepted label. Those two quotes can look comparable while representing different work.