Research-backed calculator guide

Synthetic LLM Evaluation Data Cost: Price Generation, Filtering and Human Validation

Calculate the cost of synthetic LLM evaluation data by separating generation tokens, filtering waste, judge runs, rejection rates, and human validation.

Synthetic evaluation data is not free just because a model can produce it quickly. A defensible estimate includes the tokens used to generate candidate cases, the candidates discarded by rules or a judge, any evaluator calls used to filter or score them, and the human time needed to validate the accepted set. The useful unit is usually cost per accepted synthetic evaluation example, not cost per model request.

What belongs in an LLM synthetic test-data budget?

Define the unit before you choose a model. An example might be a user input alone, an input plus reference answer, a multi-turn conversation, or a complete tool-use trace. A case with a long context and expected JSON output has a different token shape from a short classification prompt. Do not compare their unit costs without normalizing the unit.

The generation line should contain the model, input tokens, output tokens, request count, and applicable service tier. OpenAI’s published pricing page separates input, cached input, and output rates per million tokens and identifies Batch processing as a 50% reduction for eligible asynchronous work. ^[1] Anthropic’s pricing documentation likewise lists model-specific input and output rates and describes a 50% Batch API discount for both token types. ^[2] These are published rates observed on 2026-11-02, not a quote for your account. Record the model alias, region, currency, contract discount, and batch eligibility alongside each assumption.

Filtering is a separate cost center. A schema check or regular expression may be nearly free in model terms but still uses engineering and compute capacity. A semantic filter may call a judge model once or several times per candidate. If the generator and judge are different models, record them separately. If a rejected candidate is retried, count the extra generation and judge tokens. Otherwise, a high-rejection workflow can look artificially cheap.

Human validation has two common shapes. In a full review, a person checks every candidate that passed automated gates. In a sampled review, a person checks a sample of accepted cases and may add targeted examples for uncovered or risky scenarios. The latter can be sensible, but it does not prove that every unreviewed case is correct. ArXiv’s survey describes synthetic-data quality around requirements such as faithfulness and diversity and discusses selective annotation as a tradeoff between expensive human annotation and cheaper but rougher LLM-based annotation. ^[5]

Finally, include pipeline overhead when the decision concerns a production process rather than a narrow provider bill. Storage, orchestration, dataset versioning, deduplication, prompt maintenance, and experiment management may be small for a pilot and material at recurring volume. Keep them as separate rows so a finance reviewer can remove or replace an assumption without hiding it inside a blended per-example rate.

A practical cost model

Let:

• A = accepted examples required.
• q = end-to-end acceptance rate from generated candidate to accepted example.
• G = candidates generated, calculated as A ÷ q.
• I = input tokens per candidate generation.
• O = output tokens per candidate generation.
• Pg and Qg = generator input and output prices per million tokens.
• F = expected filter or judge calls per candidate.
• If and Of = filter input and output tokens per call.
• Pf and Qf = filter input and output prices per million tokens.
• r = retry multiplier applied to calls that can be retried.
• H = human validation minutes per reviewed example.
• w = loaded hourly review rate.
• s = fraction of accepted examples reviewed by a human.
• P = allocated pipeline cost for the batch.

The candidate count is:

G = A ÷ q

Generation cost is:

G × ((I ÷ 1,000,000 × Pg) + (O ÷ 1,000,000 × Qg))

Filter cost is:

G × F × r × ((If ÷ 1,000,000 × Pf) + (Of ÷ 1,000,000 × Qf))

Human validation is:

A × s × H ÷ 60 × w

Then:

total batch cost = generation + filter + human validation + P

cost per accepted example = total batch cost ÷ A

This model intentionally distinguishes acceptance rate from human review rate. A case can pass an automated filter and still fail a human check. If your process has separate gates, model each gate and multiply its survival rates rather than hiding them in one unexplained percentage.

Worked cost example

Assume a team wants 1,000 accepted synthetic evaluation examples for a regression set. The cases include a user prompt and a reference answer. The team estimates that 80% of generated candidates pass deterministic schema and duplication checks, and 90% of those pass a semantic judge. The end-to-end acceptance rate is therefore 0.80 × 0.90 = 0.72, so the generation target is 1,000 ÷ 0.72 = 1,388.89, rounded up to 1,389 candidates.

The generator is assumed to use 1,100 input tokens and 350 output tokens per candidate. The judge uses 900 input tokens and 80 output tokens per call. The generator rates are a planning assumption of $0.10 per million input tokens and $0.50 per million output tokens, matching the published GPT-6 Luna rates shown on OpenAI’s pricing page when checked on 2026-11-02. ^[1] The judge rates are an illustrative $0.10 per million input tokens and $0.50 per million output tokens, not a vendor quote. Every accepted case receives six minutes of human validation at an estimated loaded rate of $60 per hour. That labor assumption is not a BLS wage quote; it is a rounded internal planning rate. For context, the BLS May 2023 national estimate for Software Quality Assurance Analysts and Testers reported a $48.94 median hourly wage and a $52.15 mean hourly wage. ^[3]

The resulting cost is $6,025.58 ÷ 1,000 = $6.03 per accepted example. The raw model and judge token subtotal is only $0.58, or less than one cent per accepted example. The human validation line dominates because the example assumes six minutes of review for every accepted case. That is not an argument to remove human review. It is a reminder to show whether your decision is constrained by tokens, reviewer capacity, or both.

If the team reviews only 20% of accepted cases, the human line becomes $1,200, and the total becomes $1,225.58, or $1.23 per accepted example. That lower figure has a different quality boundary. It should be labeled “sampled human validation,” not presented as equivalent to full review. If semantic survival falls from 90% to 75%, the candidate requirement rises to 1,000 ÷ (80% × 75%) = 1,667 candidates. The token cost rises modestly, but the larger operational effect may be more judge calls and more cases needing diagnosis.

For recurring work, run low, base, and high cases for the acceptance rate, review minutes, and retry multiplier. A two-decimal result is useful for arithmetic, not a claim of precision about future quality.

How to reduce cost without hiding quality loss

First, use deterministic checks before semantic judging. Validate JSON, required fields, allowed labels, length limits, language, and duplicate IDs before paying for an LLM judge. This does not prove factual quality, but it prevents obviously malformed cases from reaching the expensive gate.

Second, generate in batches when latency permits. Both OpenAI and Anthropic publish batch discounts, but a discounted request is useful only if its deadline and failure-handling behavior fit the dataset workflow. Recheck the vendor page on the invoice date. Do not copy an old rate into a current forecast.

Third, separate generator and validator roles when the failure mode warrants it. The same model can reproduce its own wording and miss the same blind spot. OpenAI’s model-optimization guidance recommends representative test inputs, graders, repeated evaluation, and an optimization loop rather than treating one generation run as proof of quality. ^[4] That supports a cost model with explicit judge passes and a refresh cadence.

Fourth, stratify review. Review more cases from new domains, safety-sensitive intents, rare edge cases, or low-confidence judge outcomes. Review fewer routine cases only when you retain a baseline sample and record the coverage rule. An apparently low cost per accepted example may simply be a low validation rate.

Use the AI Agent Cost Calculator when generation is one component of a larger agent workflow. If the dataset is used repeatedly in an observability program, compare judge, retention, and incident-prevention assumptions with the AI observability ROI calculator. When provider usage does not match the estimate, reconcile token quantities and service tiers with the AI API invoice reconciliation calculator.

Limitations and exclusions

This worksheet excludes the value of a better evaluation set, model-quality uplift, avoided incidents, and revenue impact. It calculates attributable preparation cost; it does not prove that synthetic data is better than human-authored data or that a lower cost creates ROI.

It also excludes a full annotation-management platform fee unless you add it to pipeline allocation. Taxes, contract discounts, reserved capacity, storage egress, staff management, privacy review, and security approval may be material. Add them when they are part of the decision boundary.

Published rates are time-sensitive. The example uses USD and rates checked on 2026-11-02. Model names, prices, batch eligibility, tokenization, and product availability can change. Use invoice data or an account-specific quote for approval. The BLS wage data is from May 2023, so it is a context point for labor scale, not a current contractor quote or a universal loaded rate.

FAQ

What is the right denominator for synthetic evaluation data cost?

Use the number of examples that pass the defined quality gates and are actually admitted to the evaluation set. Keep generated candidates, automated survivors, and human-approved examples as separate counts. If your process admits cases without human review, call the metric “cost per accepted automated case” rather than implying that every case received human approval.

Should rejected synthetic examples be included in the cost?

Yes. Rejected candidates consumed generation, filtering, storage, and sometimes reviewer time. They belong in the numerator and stay out of the accepted denominator. If rejected cases are regenerated, include the additional attempts. A workflow that discards 40% of candidates is not priced correctly by dividing only the final API bill by the accepted rows without showing the rejection funnel.

Is human validation required for an LLM evaluation dataset?

Not for every low-risk use case, but it is a quality decision that must be explicit. Automated checks can catch structure, duplication, and some semantic problems. Human review is useful for subtle ambiguity, unsafe assumptions, domain nuance, and missing edge cases. If you review a sample, publish the sampling rule and do not describe the entire set as fully human validated.

How do I calculate the cost per synthetic test case?

Calculate the required candidate count from the acceptance rate, price generation and filter tokens separately, add retries and human time, then divide the total by accepted cases. For example, 1,389 candidates × generator cost plus 1,389 judge calls × judge cost plus 1,000 accepted cases × review cost per case, then add pipeline overhead and divide by 1,000.

Should the generator and evaluator use different models?

Often, yes, when independence improves confidence or the evaluator needs a different capability. It is not a universal rule. A different model adds another price and token shape, so model it as a separate judge pass. If you use the same model for both roles, document that limitation and use human calibration or an independent audit sample where the result matters.

How often should a synthetic evaluation set be refreshed?

Refresh when the product, prompt, retrieval corpus, model, policy, or observed traffic distribution changes. A recurring refresh budget should include new candidate generation, deduplication, validation, and version comparison. Do not assume that reusing an old set is free: it can hide drift and create a maintenance cost that belongs in the operating model.