Research-backed calculator guide

LLM Evaluation Budget by Traffic Slice: How Much Should You Score?

Calculate LLM evaluation sampling cost by traffic slice, evaluator depth, retries, and risk so production coverage is affordable and auditable.

What to count before setting a coverage rate

A production trace is not always one evaluation. A simple chatbot may need one final-answer check. An agent trace can expose several scoring surfaces: retrieval relevance, route selection, tool arguments, policy compliance, and final task success. Langfuse describes this distinction by supporting evaluator targets at the observation level and by filtering which observations receive a rule. That lets a team evaluate a final generation without automatically judging every child operation. ^[1]

Define a traffic slice by a property that changes risk, cost, or required evidence. Useful dimensions include workflow, model route, geography, customer plan, tool type, release version, or a known failure mode. Do not create slices merely because a dashboard can. Every slice should answer a decision question such as “Did the new retrieval prompt harm billing-support answers?” or “Are premium customers receiving safe tool calls?”

For each slice, capture these inputs:

1. Eligible traces. Count requests or observations that could be scored during the budget period. Use the same unit throughout. If one trace contains five candidate observations, do not silently compare it with a request count.
2. Sampling rate. This is the fraction of eligible units sent to an evaluator. It can be 100% for a small critical stream or 2–10% for a large routine stream.
3. Evaluation surfaces. Count the distinct operations being scored. Instrumentation is not the same as evaluation: a recorded tool call may cost storage but zero judge calls until an evaluator runs.
4. Judge passes. One correctness judge and one groundedness judge are two passes over the same sampled unit. A pairwise comparison or second independent judge adds another pass.
5. Retry factor. A factor of 1.05 means the design expects five extra judge attempts per 100 successful evaluations. Include provider errors, timeouts, and deliberate re-runs rather than hiding them in a contingency line.
6. Token shape and price. Judge prompts often include the user input, application output, rubric, and sometimes retrieved context. Estimate input and output tokens separately. A visible answer alone is not the whole judge prompt.
7. Escalation and retention. Human review, annotations, trace retention, and data processing are not judge-token costs. Keep them separate so assumptions can be changed without rebuilding the model.

A model price is a vendor quote only when you have a current quote or published rate for the exact account and service tier. The worked example below uses a published list-price assumption from OpenAI's public pricing page: GPT-6 Luna is shown at $0.10 per 1 million input tokens and $0.50 per 1 million output tokens, with a separate cached-input rate. ^[2] Treat those rates as dated assumptions, not a promise that your provider, region, contract, or model alias will match them.

A defensible formula for an online evaluation budget

Let slice i have eligible traces Tᵢ, sampling rate sᵢ, surfaces eᵢ, judge passes pᵢ, and retry factor rᵢ. Its expected judge calls are:

Cᵢ = Tᵢ × sᵢ × eᵢ × pᵢ × rᵢ

If a slice uses more than one judge model, calculate each model as a separate pass. This matters because a cheap structural check and an expensive reasoning judge do not have the same unit cost. Likewise, a code evaluator may have near-zero model-token spend while still consuming compute or engineering capacity.

For judge model j, with input tokens Iⱼ, output tokens Oⱼ, input price Pᵢⱼ, and output price Pₒⱼ, the slice cost is:

judge costᵢⱼ = Cᵢ × ((Iⱼ ÷ 1,000,000 × Pᵢⱼ) + (Oⱼ ÷ 1,000,000 × Pₒⱼ))

Sum the model costs across judges. Then add human escalation:

human cost = escalated cases × loaded review cost per case

The loaded review assumption should include the compensation or internal time rate you actually use. It is not a model-provider price. If you do not know it yet, label it as an estimate and run a low/base/high sensitivity case.

This calculation is more useful than multiplying “monthly requests × one judge price” because it makes depth visible. A 10% sample can still be expensive when each trace has four surfaces and two judges. A 50% sample can be cheaper when it evaluates only a short final answer with one lightweight pass.

Worked cost example

Assume a support agent produces 100,000 eligible traces in one month. The team wants broad monitoring for routine traffic, denser coverage for a risk-sensitive workflow, and full coverage for a small critical workflow. The judge is assumed to receive 1,200 input tokens and return 120 output tokens per pass. The illustrative judge rates are $0.10 per million input tokens and $0.50 per million output tokens, based on the dated OpenAI list-price assumption above. OpenAI also documents Batch processing as a 50% reduction for eligible asynchronous work, but this example does not apply that discount because it assumes online monitoring. ^[2]

The per-call arithmetic is (1,200 ÷ 1,000,000 × $0.10) + (120 ÷ 1,000,000 × $0.50) = $0.00018. Multiplying $0.00018 by 28,375 expected calls gives $5.1075, rounded to $5.11. The critical slice consumes 58.1% of judge calls even though it is only 5% of eligible traces. That is not a mistake. It is the cost of choosing three scoring surfaces and a 10% retry allowance for the highest-consequence path.

Now add a separate human-review assumption. Suppose 2% of critical-slice evaluations escalate: 16,500 × 2% = 330 cases. At an estimated loaded review cost of $2.50 per case, human review is $825. This is an internal planning assumption, not a vendor quote. The example's judge-only budget is $5.11, while its judge-plus-human budget is $830.11 before storage, pipeline, and staff coordination.

That contrast is the decision insight. Token spend can be tiny while review operations dominate. Conversely, a long rubric or expensive judge can make model spend meaningful even when human escalation is rare. Present both layers to finance and engineering rather than hiding everything inside “evaluation cost.”

How to choose sample rates by risk

Use a baseline sample for every important slice, then spend additional coverage where a miss is costly or where the system is changing. A practical starting rubric is:

• Routine: sample enough traffic to detect broad quality drift, and periodically audit the sampled path against a stronger judge or human label.
• Risk-sensitive: increase sampling when the workflow affects money, access, customer commitments, or retrieval-grounded advice. Add a second surface when the final answer alone cannot explain failure.
• Critical: consider full coverage for a small stream, deterministic checks for hard constraints, and human escalation for ambiguous or consequential cases.

These are planning categories, not universal rates. Choose a rate from the error you can tolerate, the volume you can afford, and how quickly you need a signal. If a slice is too small for stable percentages, use an explicit minimum count, such as “at least 200 observations per day,” and document the resulting monthly ceiling.

Do not confuse evaluation coverage with statistical certainty. A 10% sample can be useful for trend detection, but it may miss a rare failure concentrated in one country, customer tier, or tool route. Stratify or oversample those slices when the risk is uneven. Keep the original denominator so the dashboard reports both raw sample counts and weighted coverage.

OpenAI's eval documentation frames an eval as a task definition, test inputs, graders, a run, and analysis of the results. That structure is a useful reminder that a score has an operating lifecycle: dataset or traffic selection, execution, grading, and iteration. ^[3] For production monitoring, add version, slice, evaluator, and model identifiers to every result so a change in coverage is distinguishable from a change in quality.

Reduce cost without making the score meaningless

Start with code checks for deterministic properties such as JSON validity, required fields, allowed tool names, or latency thresholds. Use a lightweight judge for routine semantic checks. Reserve a stronger judge for high-uncertainty or high-consequence cases. Then audit a sample of the cheap path with a stronger judge or human labels. Layering preserves a quality-control signal while keeping expensive passes selective.

Use observation-level targeting when the whole trace is not needed. Langfuse notes that observation-level evaluators can target individual operations and combine observation and trace filters. That supports rules such as “evaluate final generations for all premium users” or “evaluate retrieval observations only for the new release.” ^[1]

Cache stable rubric text when the provider offers cached-input pricing, but do not assume all prompt tokens qualify. Separate ordinary input, cached input, output, and any long-context tier in the worksheet. For asynchronous regression runs, check whether a provider's batch product is available and whether its deadline is compatible with your release process. OpenAI's pricing page lists separate processing modes and dates them on the live page; recheck before using any discount in a budget. ^[2]

Finally, cap the downside. Set a monthly call ceiling, an alert when retry factors rise, and a stop or fallback policy for evaluator failures. A budget that assumes every evaluator call succeeds can fail exactly when production is unstable and retries multiply.

For the next step, reconcile the estimate to actual provider usage with the AI API invoice reconciliation calculator. If you are measuring whether monitoring paid for itself, compare prevented incident and triage costs in the AI observability ROI calculator. For agent workflows with hard spending limits, pair this worksheet with the AI agent budget guardrail and P95 calculator.

FAQ

Should I evaluate 100% of production traffic?

Not automatically. Full coverage is sensible when the slice is small, the consequence of a miss is high, or the evaluator is inexpensive and reliable. High-volume routine traffic often benefits from sampling plus targeted filters. Keep a baseline random sample so the system can detect mix changes, then add denser coverage for critical workflows and new releases.

What is the difference between evaluation coverage and judge spend?

Coverage is how much eligible traffic receives a score. Judge spend is the token cost of producing those scores. Two systems can have the same coverage but different spend because one scores one surface with a short prompt and the other runs multiple long judges. Report eligible volume, sampled volume, surfaces, passes, and cost together.

How should retries appear in an online evaluation budget?

Use an explicit retry factor. If 1,000 successful evaluations require 1,050 attempts on average, use 1.05, not a hidden contingency percentage. Track retries by provider, evaluator, and slice. A rising retry factor can indicate rate limiting or an evaluator defect, and it can make a previously safe budget exceed its cap.

Do human labels belong in LLM evaluation sampling cost?

They belong in the total evaluation budget, but not in judge-token cost. Keep human review as a separate line with an explicit escalation rate and loaded cost per case. This makes it possible to compare a stronger judge, a larger random sample, or more human calibration without conflating unlike units.

How many evaluation surfaces should one trace have?

Only as many as are tied to a decision. A final-response score cannot diagnose a bad retrieval or malformed tool argument, but scoring every intermediate operation can multiply cost without improving actionability. Start with the surface most connected to the risk, then add a targeted surface when a failure mode remains invisible.

What should I do when a slice is too small for a percentage?

Set a minimum observation count and a maximum budget. For example, evaluate at least 200 observations per week if available, but never exceed a defined monthly call cap. Record the actual denominator and sample count so a 100% score on 40 observations is not presented as equivalent evidence to 100% on 4,000.

Are published model prices enough for a finance-approved budget?

They are a starting assumption, not always a final quote. Confirm model alias, region, service tier, cached-input eligibility, batch eligibility, taxes, contract discounts, and any platform fee. Date every price in the worksheet. If those details are unknown, present low, base, and high cases instead of a single false-precision number.