Research-backed calculator guide

LLM Classification Model Selection: Macro-F1, Cost per Correct Label and p95 Latency

A practical LLM classification benchmark that compares macro-F1, cost per correct label, and p95 latency with dated pricing, reproducible assumptions, and failure-mode checks.

What this benchmark should measure

A fixed-label classifier maps an input to one label from a known set, such as billing, technical_support, or account_access. The benchmark should test the behavior that the production system actually needs. If an input may belong to none of the known labels, add an abstain or out-of-scope class instead of silently forcing a choice. Research on classification evaluation has shown why label-space design matters: the Know-No study tests cases where the gold label is absent and argues that ordinary accuracy can overstate model behavior in those situations. That is a useful warning to include open-set or abstention cases when they exist in your product. ^[1]

Macro-F1 is the quality gate when minority classes matter. For each class, calculate precision and recall, combine them into that class's F1, and take the arithmetic mean across classes. This gives a small class the same weight as a large class. Overall accuracy remains useful as a separate operational statistic, but it can hide a classifier that predicts the dominant class too often. If your business cost is asymmetric, also report a cost-weighted confusion matrix or a per-class recall floor; macro-F1 is not a substitute for a safety or service requirement.

The evaluator workflow in Hugging Face's documentation illustrates a practical baseline: a text-classification evaluation can use a labeled dataset, a model pipeline, an explicit label mapping, and multiple metrics. Its example reports a requested quality metric alongside latency, samples per second, and total time, while warning that measured time includes pipeline processing and depends on hardware and batch size. ^[2] For an LLM API, extend that idea by recording network time, queueing, retries, output parsing, and the time until a valid class is available.

Cost per correct label connects quality to spend. For a benchmark with accepted predictions, use:

total benchmark cost = input cost + output cost + cache/tool charges + retry cost
cost per correct label = total benchmark cost / number of correct accepted labels

If you want a cost-per-successful-task number rather than a cost-per-correct-label number, define “successful” separately. For example, a task might require the correct label, a confidence field, and valid JSON. A correct label inside malformed JSON is not successful if the production parser rejects it. Keep both metrics when a downstream system has strict acceptance rules.

p95 latency is the time at or below which 95% of measured requests finish. For classification, define the endpoint clearly: request start to a parsed, valid label, including queueing and network time if those are part of the user experience. A local model can report inference-only p95 as well, but do not compare it directly with a hosted API's client-visible p95 unless you disclose the boundary. For one-label classification, time to a complete valid response is usually more decision-relevant than streaming smoothness.

Build a reproducible test set

Start with a stratified sample rather than a random sample alone. Include the production class mix, but ensure every class has enough examples to make its F1 interpretable. Add difficult examples, short and long inputs, spelling variation, multilingual or code-switched text if relevant, and cases that should be abstained. Keep a locked test set separate from prompt tuning. If labels are disputed, adjudicate them before the model run and preserve the labeling rule.

Freeze the instruction and output contract. A fair prompt states the label names, tells the model to return one allowed label, and defines what to do when the input is ambiguous. Use the same label descriptions across candidates unless the candidate requires a documented adapter. Record the model ID, API endpoint, system prompt, temperature or equivalent sampling setting, maximum output tokens, stop condition, and parser version.

For each request, log a row with an anonymized case ID, gold label, raw response, parsed label, validity flag, retry count, input tokens, output tokens, request start and end timestamps, status code, and error category. Do not discard invalid responses before computing the failure rate. An invalid label is a production outcome that may trigger a retry, human review, or fallback call.

Run enough repetitions to estimate the tail. A single serial pass can describe one moment, not a reliable p95 under load. Use a concurrency level that resembles the intended workload and run a warm-up before timed measurements. Keep the batch size, connection reuse, region, timeout, and rate-limit behavior fixed. If the provider's service tier changes, make it a separate benchmark arm rather than blending its timings with standard processing.

The cost formulas

For a standard text request, normalize provider rates to dollars per one million tokens:

input_cost  = input_tokens  / 1,000,000 × input_rate
output_cost = output_tokens / 1,000,000 × output_rate
request_cost = input_cost + output_cost

For a benchmark with many requests, sum the actual token categories. Do not multiply an average by request count if the distribution is highly skewed; the sum of recorded usage is safer.

benchmark_cost = Σ(input_cost + output_cost + cache_cost + tool_cost)
                + retry and fallback costs
cost_per_correct_label = benchmark_cost / correct_accepted_labels

OpenAI's current pricing page separates input, cached input, and output prices and publishes rates per one million tokens. ^[3] Anthropic likewise separates base input, cache writes, cache hits, and output, and defines MTok as million tokens. ^[4] Google documents input, output, context caching, and model-specific treatment of thinking tokens, with rates that can change by tier or date. ^[5] These are vendor rate-card observations checked on 2026-11-02, not a quote for your account. Verify the exact model ID, endpoint, processing mode, region, credits, taxes, and contract terms before a purchase decision.

If a classifier uses a stable prefix, split fresh input, cache writes, and cache hits rather than applying one blended input rate. If a model exposes reasoning or thinking tokens and bills them, include them in the priced output category or in the provider-specific field that represents them. If a request fails after consuming tokens, count the charge when the provider bills it. If your application retries, count the retry as real spend even when the final label is correct.

Worked cost example

The table below is an original, illustrative benchmark. It is not a vendor performance claim. Assume 100,000 labeled English support tickets, 250 input tokens per request, and 3 output tokens for a constrained label response. Each candidate uses standard synchronous text processing, no cache, no tools, no media, no taxes, no credits, and one measured attempt per request. The example rate cards were checked on 2026-11-02 and must be rechecked before use.

The arithmetic is straightforward. For candidate A, 25 million input tokens cost 25 × $0.10 = $2.50; 0.3 million output tokens cost 0.3 × $0.50 = $0.15; total cost is $2.65. Dividing $2.65 by 91,000 correct labels gives approximately $0.0000291 per correct label, or about $29.12 per million correct labels under these assumptions. Candidate C costs much more and is slower in this scenario, while also having lower assumed exact accuracy. It is dominated here, but that conclusion is only for the stated workload and measured quality.

The table intentionally keeps macro-F1 and exact correct labels separate. Macro-F1 is the class-balanced quality score; the denominator for cost per correct label is the count of exact accepted predictions. If the classes are imbalanced, two candidates can have similar accuracy but different macro-F1. Inspect each class's recall before approving a model. If 9,000 of the 100,000 examples belong to a high-risk minority class, add a minimum recall constraint rather than allowing a low-cost majority-class strategy to win.

The rate inputs are representative placeholders tied to the published unit structure, not claims about these candidates' measured quality. OpenAI's published rate card currently shows a low-cost model at $0.10 per million input tokens and $0.50 per million output tokens, while the Google pricing page shows model- and tier-specific prices and scheduled changes. ^{[3] [5]} The correct production practice is to replace the illustrative rows with the exact model IDs you tested and the usage amounts returned by the provider.

Make the decision with gates and a frontier

Set hard gates before looking at the winner. A sample policy might require macro-F1 of at least 0.90, p95 of no more than 500 milliseconds, invalid-label rate below 0.5%, and a minimum recall for the two most sensitive classes. These are example assumptions, not universal thresholds. If no candidate passes, do not hide the miss by ranking the least-bad option; change the model, prompt, infrastructure, label policy, or service target.

After gating, plot or tabulate cost per correct label against p95 latency and macro-F1. A candidate is Pareto-dominated if another candidate is no more expensive, no slower, and at least as good in quality, with one strict improvement. The cost-aware classification study found that framing selection as a multi-objective problem can expose trade-offs between fine-tuned encoders and prompted LLMs across macro-F1, latency, and monetary cost. ^[1] Its result is evidence for using this framework, not a reason to copy its model ranking to a different dataset or provider environment.

For an API classifier, the cheapest token rate may lose after retries, long prompts, or review. For a self-hosted encoder, include GPU or CPU allocation, idle capacity, batching, quantization validation, deployment operations, and the fraction of capacity reserved for peaks. The Self-hosted GPU inference cost calculator is useful when infrastructure is part of the comparison. For API-only scenarios, normalize the rate card with the Model pricing comparison, then reconcile measured token counts in the AI token cost calculator. A multi-step fallback chain belongs in the LLM cost calculator rather than a one-call estimate.

Common benchmark errors

Comparing different prompts. A large model with a detailed label glossary and a small model with label names only is not a controlled comparison. Keep the task information equivalent and document any adapter.

Using mean latency as the SLA. Mean and p50 can look healthy while queueing or rate limits produce a bad p95. Report p50, p95, sample count, concurrency, and timeout policy together.

Counting only visible output tokens. Tool definitions, system prompts, retrieved context, cached tokens, and provider-specific thinking tokens can affect billing. Use response usage metadata whenever available.

Treating invalid output as correct. A response that says “probably billing” when the parser accepts only billing is invalid unless the contract explicitly allows that format. Count repair calls and human review in the economics.

Pooling class scores incorrectly. Macro-F1 is the mean of class-level F1 values, not the F1 obtained after pooling all predictions. Report weighted or micro metrics separately when they answer a different operational question.

Mixing hardware boundaries. A local p95 that excludes queueing cannot be compared directly with an API p95 measured from the client. State whether timing includes serialization, network, queue, generation, parsing, and retry.

Treating a dated price as permanent. Provider pricing pages change, models retire, and discounts expire. Save the checked date and model ID beside every benchmark result. The Model pricing comparison guide explains why a rate-card estimate is a planning number, not an invoice forecast.

FAQ

Is macro-F1 better than accuracy for LLM classification?

Not universally. Macro-F1 is usually more informative when every class matters or the class distribution is imbalanced because it gives each class equal weight. Accuracy is useful when each example has the same business consequence and the test distribution matches production. Report both, plus per-class recall or a cost-weighted confusion matrix when errors have different consequences.

What is cost per correct label?

It is total benchmark spend divided by the number of correct, accepted labels. Total spend should include billed input and output tokens, cache or tool charges, retries, fallbacks, and any other workload-specific costs. The denominator should exclude invalid or incorrect labels. If the downstream task needs valid JSON or human approval, also calculate cost per successful accepted task with that stricter definition.

Should classification latency use p95 or p99?

Use the percentile that matches your service objective. p95 describes the slower tail for 5% of requests and is a practical default for many interactive services. p99 is appropriate when rare delays are materially harmful. Always disclose the timing boundary, concurrency, sample size, warm-up, region, and whether the measurement ends at the first token or the final parsed label.

How many examples do I need for a useful benchmark?

There is no single correct number. Start with enough examples to represent the production distribution and give each important class a stable estimate. Add a separate edge-case and regression set. Report per-class support and uncertainty rather than presenting a precise-looking macro-F1 from a tiny sample. If a class is rare but high risk, oversample it for diagnosis while preserving a production-weighted score for capacity planning.

Can a fine-tuned encoder beat a prompted LLM on cost and latency?

It can for a stable fixed-label task, but do not assume it will. Fine-tuned encoders may have low per-request inference cost and latency, while prompted LLMs may reduce labeling or engineering effort and handle changing instructions more flexibly. Benchmark both on the same locked set, include training and deployment costs where relevant, and preserve an abstention or fallback path for inputs outside the encoder's training scope.

How often should this benchmark be rerun?

Rerun when the model ID, prompt, tokenizer, provider price, endpoint, region, traffic shape, label policy, or production class mix changes. For a stable service, a scheduled regression run can detect quality and p95 drift. Reconcile benchmark predictions with provider usage and production outcomes before changing the selected model.