Research-backed calculator guide

Human Calibration Set Size for LLM Judges: A Cost-Quality Decision Rule

Choose an LLM judge calibration sample size by balancing class coverage, uncertainty, human review cost, judge tokens, retries, and escalation.

What the calibration set is buying you

Calibration is not primarily about making a dashboard number look stable. It is about estimating whether the judge’s decisions are useful for a specific operational action. That action might be blocking a release, routing a trace to a human, selecting a prompt, or reporting a safety rate.

A calibration item therefore needs at least the production input, the output being judged, the frozen rubric version, the judge result, and a human reference label or adjudicated decision. Store the model and prompt versions too. If the judge sees retrieved context or tool-call metadata in production, include the same fields in calibration. Otherwise, you are measuring a different evaluator than the one you intend to deploy.

Separate these questions:

1. Can humans apply the rubric consistently? If two domain reviewers disagree frequently, a judge cannot be expected to match a single supposedly perfect label.
2. Does the judge agree with the human reference? Measure raw agreement, but add a chance-adjusted metric such as kappa when the label setup supports it.
3. Where does it fail? Inspect false positives, false negatives, per-class results, and disagreement reasons. The operational cost of one missed critical failure can outweigh a higher overall agreement rate.
4. Will the result transfer? A balanced challenge set can test rare failures, but it does not estimate production prevalence. Keep a natural-distribution holdout alongside any enriched failure set.

The academic reporting work on LLM-as-a-judge evaluations makes the same statistical point in a more formal setting: calibration labels estimate judge sensitivity and specificity, and uncertainty in those calibration estimates contributes to the uncertainty of the corrected production estimate.^[3] In other words, increasing the production test set does not eliminate uncertainty created by a tiny or weak calibration set.

A decision rule for LLM judge calibration sample size

Use the following sequence before choosing a number.

Gate 1: Set the unit and the decision

Decide whether one item is a response, conversation, trace, tool trajectory, or pairwise comparison. Do not mix turns from the same conversation as if they were independent if the decision is made at conversation level. Define the label type as binary, categorical, ordinal, or continuous. Then name the decision threshold: for example, “block a release if the unsafe-output rate exceeds 2%” or “accept a prompt only if groundedness agreement is at least as good as the previous version.”

The threshold determines what evidence matters. If the release decision depends on the unsafe class, total agreement is not enough. If the task is pairwise model selection, calibrate on both preference directions and randomize presentation order. If the judge produces a 1–5 score, decide whether you need weighted agreement, error within one point, or a binary pass/fail transformation.

Gate 2: Set class and slice minimums

Choose the minimum count for every class or slice that can change the decision. Suppose a production failure rate is 5%. A random sample of 200 items has an expected average of only 10 failures. The estimate of failure recall will be unstable if a few cases change the numerator. Enrich the calibration set with failures when necessary, but label the set as enriched and use a natural-distribution set for prevalence estimates.

A useful starting plan is 30–50 representative items for rubric iteration, with multiple human annotations on the difficult or high-risk subset. Move to 100 or more when you need a directional agreement estimate. Move toward 385 when a simple overall proportion needs approximately ±5 percentage-point precision at 95% confidence. For a rare class, calculate the count requirement for that class separately; the total n is not a substitute for class coverage.

Gate 3: Set the uncertainty target

For a rough binary proportion, use:

n ≈ z² × p(1 − p) ÷ E²

For a 95% planning estimate, z is 1.96. When the expected proportion p is unknown, use 0.5 because it produces the largest variance. E is the desired half-width of the interval. With p = 0.5 and E = 0.10, n is approximately 96, commonly rounded to 100. With E = 0.05, n is approximately 385.

Do not present that calculation as a universal kappa or F1 sample-size formula. Kappa depends on prevalence and marginal label behavior. F1 depends on the number of actual and predicted positives. Multiple annotations add another source of dependence. The academic method cited above explicitly carries both test-set and calibration-set terms into its confidence interval and shows that calibration allocation can be asymmetric when the two label types contribute different amounts of uncertainty.^[3]

Gate 4: Add a risk multiplier and a change trigger

High-stakes, fast-changing, or high-cost-to-miss rubrics need more evidence than a stable style rubric. Increase the set when the judge model changes, the rubric changes, the generator model changes, a new user segment appears, or production disagreement patterns shift. Reuse a fixed holdout for before-and-after comparisons, but refresh part of the calibration pool with current traffic so that drift is visible.

A simple policy can be: pilot at 40 items; retain 120 items for a normal production rubric; use 200–400 when the release decision needs tighter uncertainty or several important classes; and trigger a new review when a major version changes or the confidence interval crosses the operating threshold. The numbers are policy defaults, not vendor requirements. Document why your risk, prevalence, and review capacity justify them.

The cost model

The judge call is often the smallest part of a calibration budget. Human labeling, adjudication, and the opportunity cost of repeatedly reviewing hard cases can dominate.

Use this model for one calibration cycle:

calibration cost = judge calls
                 + independent human labels
                 + escalation or adjudication
                 + platform and storage allocation

judge calls = (production evaluations + calibration evaluations)
              × repeat rate
              × judge token cost per call

human labels = calibration items × reviewers per item
               × minutes per label ÷ 60 × hourly reviewer rate

escalation = calibration items × escalation rate
             × extra minutes ÷ 60 × hourly reviewer rate

Count judge calls separately from human labels. A “120-item calibration set” may mean 120 human-reviewed items but 240 judge calls if the evaluator is run twice, and more if you compare two prompt versions. If a trace produces multiple score dimensions, count each score call or each billed evaluation unit according to the actual system meter.

Use the LLM observability cost calculator to organize trace, score, retention, and evaluation units. Use an LLM cost calculator for a provider-specific token estimate once you have a dated rate card. Do not turn the example below into a vendor quote.

Worked cost example: 120-item calibration set

Assume a support agent generates 10,000 candidate outputs per month. The team evaluates 20%, or 2,000 outputs. It keeps a 120-item calibration set with two independent human labels per item. Each label takes three minutes. Review labor is modeled at an illustrative $40 per hour. Twelve percent of calibration items are escalated to one extra reviewer for three minutes.

For the judge, assume 1,200 input tokens and 200 output tokens per call. To make the arithmetic inspectable, use illustrative rates of $0.50 per million input tokens and $2.00 per million output tokens. These are assumptions, not a current provider quote and not a recommendation of a model.

The recurring judge spend is only about $2.33 under these assumptions, while human calibration is $510. If the team spreads the cycle across 2,000 evaluated outputs, the fully loaded calibration allocation is about $0.27 per evaluated output. The marginal judge cost, including the 10% retry allowance, is about $0.0011 per production evaluation. Those are different metrics and should not be conflated.

Now compare a 40-item pilot using the same two reviewers, three minutes per label, 12% escalation, and $20 fixed allocation. Primary labels cost $160, escalation is approximately $10, and judge spend is roughly $2.15 when 10% retries are included. The pilot is about $192.15. It is cheaper, but it has less class coverage and a wider uncertainty range. Pay for 120 items when the decision is consequential, the failure class is sparse, or the cost of a wrong release decision is high. Keep the 40-item version for prompt iteration, not as permanent evidence of production reliability.

How to make the sample more informative without simply making it larger

Stratify the set by the dimensions that can change judge behavior: label class, traffic route, model version, language, customer segment, output length, judge confidence, and recent releases. Keep a natural-distribution slice for prevalence and a targeted slice for rare failures. Record the sampling fractions so you do not mistake an enriched set for production performance.

Ask humans to label independently before seeing the judge verdict when practical. Adjudicate disagreements rather than silently selecting whichever label is convenient. Preserve the reason for each correction. Those reasons can expose a vague rubric, missing context, a systematic position bias, or a genuine model limitation.

Run the judge on the same frozen items before and after a prompt or model change. LangChain’s described workflow uses corrections, representative few-shot examples, and agreement tracking to make evaluator alignment measurable over time.^[1] Langfuse similarly recommends testing an evaluator on representative sample observations and making the input, output, reference, score definition, and model mapping explicit.^[2]

For binary labels, report a confusion matrix and per-class precision and recall. For multiple annotators, report how human agreement was established and how ties were handled. If the judge is nondeterministic, repeat a subset and measure self-consistency. A larger sample of one noisy run does not necessarily replace repeated runs on the same hard items.

Assumptions, exclusions, and limitations

This decision rule assumes that calibration items are sufficiently independent for the chosen uncertainty approximation. Items from the same conversation, customer, or incident may be correlated, so sample at the decision unit and use group-aware resampling when estimating uncertainty. It assumes reviewers are qualified for the rubric and that the rubric remains frozen during a comparison.

The worked example excludes tax, negotiated discounts, annotation-platform minimums, reviewer recruitment, training, engineering time, model-hosting overhead, data egress, and the cost of the production model that generated the outputs. It also assumes one judge score per output and treats retries as a simple 10% allowance. Replace those assumptions with observed token counts, actual reviewer rates, and the billing units of your stack.

This is not a claim that 100, 120, 200, or 385 items will guarantee a target kappa, F1, release outcome, or agreement with every expert. The required size depends on prevalence, label structure, evaluator variance, number of annotators, desired precision, and the cost of missed errors. The point is to make the tradeoff explicit and reproducible.

FAQ

Is 50 examples enough to calibrate an LLM judge?

Fifty representative examples can be enough for early rubric iteration and to discover obvious disagreement patterns. It is usually weak evidence for a stable production estimate, especially when important failures are rare or the judge has several output classes. Treat 50 as a pilot unless your documented risk and uncertainty analysis supports using it as a final set.

How many human labels should each calibration item receive?

Use at least two independent labels for a meaningful subset when human agreement is not already established. Add a third reviewer or adjudicator for disagreements and high-risk cases. One label per item can be operationally cheaper, but it leaves reference-label reliability unknown. The right choice depends on the decision risk, reviewer expertise, and whether the rubric is nominal, ordinal, or continuous.

Should the calibration set match production class balance?

Keep one slice that matches production when estimating real-world prevalence and another enriched slice when testing rare failures. A balanced challenge set gives useful coverage but can distort production precision and overall rates. Store the sampling design and report results by class instead of publishing one aggregate number.

How often should an LLM judge be recalibrated?

Recheck it after a judge-model, generator-model, rubric, prompt, retrieval, policy, or major traffic-segment change. For a stable production workflow, schedule a recurring review rather than waiting for a visible incident. Compare the unchanged holdout over time and refresh a portion with current production examples so drift is measurable.

Does a larger production test set replace a larger calibration set?

No. More production evaluations reduce uncertainty about the judge’s observed output rate, but they do not remove bias or uncertainty in the human-calibrated sensitivity and specificity. The reporting paper’s estimator and confidence interval explicitly include both test and calibration sample sizes.^[3] If the calibration set is weak, millions of automated scores can make a wrong estimate look more precise.

How should calibration cost be allocated to each evaluated output?

Separate fixed and marginal costs. Divide the human calibration cycle, escalation, and platform allocation by the number of outputs or decisions that benefit from the calibration period. Separately report the marginal judge token cost, including expected retries. This prevents a low token price from hiding a large human review program and makes budget changes easier to explain.

Related tools

Use the AI observability ROI calculator to connect evaluation spend to incident and triage outcomes. Use the AI API invoice reconciliation calculator to compare modeled judge calls with actual usage rows. For a broader spend ceiling, use the AI agent budget guardrail and P95 calculator.