Research-backed calculator guide

Reranker API vs Self-Hosted: Break-Even by Throughput and SLO

Compare reranker API spend with self-hosted GPU TCO using candidate count, chunking, utilization, redundancy, throughput, and a p95 latency target.

What you are actually comparing

A reranker is a second-stage scorer. First-stage keyword, vector, or hybrid retrieval produces a shortlist. A cross-encoder then reads the query with each candidate and returns relevance scores, after which the application keeps the best few documents. Cohere describes Rerank as sorting text inputs by semantic relevance on top of an existing search solution ^[3]. NVIDIA describes the same pipeline boundary: a lighter embedding stage finds a pool, while a more accurate reranker narrows that pool before generation ^[5].

That boundary matters for cost. A request with 20 candidates is not equivalent to a request with 100 candidates. A 100-document request is also not equivalent to one in which long documents are split into 240 billable chunks. Count the query-document pairs that the serving system actually scores, not only user questions.

The two options also expose different risk:

• API: variable spend, provider availability and rate limits, a simple integration boundary, and less responsibility for GPU scheduling and model serving.
• Self-hosted: a mostly fixed capacity bill, control over model and data location, and responsibility for model downloads, autoscaling, queueing, patching, observability, failover, and idle capacity.

NVIDIA’s retrieval guidance explicitly frames accuracy, latency, and throughput as simultaneous design considerations. Its production advice also distinguishes low-volume workloads from high-volume, high-velocity pipelines and recommends evaluating the chosen model on the local dataset rather than assuming a benchmark transfers unchanged ^[5]. That is the right standard for a break-even decision.

A provider-neutral break-even model

Let the following inputs describe one month:

• Q: user rerank requests.
• D: average documents submitted per request.
• C: average chunks per submitted document after any provider or application splitting.
• R: retry multiplier. For example, 1.03 represents 3% extra attempts.
• P: API price per billable search unit.
• G: number of paid self-hosted GPUs or instances.
• H: paid hours per month.
• B: hourly price per GPU or instance.
• O: monthly non-GPU overhead, such as CPU, storage, network, monitoring, and support.
• E: monthly engineering and operations allocation for the self-hosted service.
• F: monthly failover or standby cost that is not already included in G or O.
• T: measured cross-encoder pair throughput per GPU at the target SLO.
• U: productive utilization credited to the reranker workload.

If the API bills one search for up to 100 documents, the approximate billable search count is:

API units = Q × R × ceiling(D × C ÷ 100)

Use the provider’s exact rule instead of this example. If the provider bills per document, token, or a tiered request, replace the ceiling expression with that meter. Cohere’s pricing page is particularly important for long documents because it says that chunked documents count toward the total documents ranked ^[1].

The API estimate is then:

API cost = API units × P

For self-hosting, first calculate the monthly pair demand:

Required pairs/month = Q × D × C × R

Then calculate measured pair capacity:

Credited pairs/month = T × 3,600 × H × G × U

The configuration passes the capacity screen only if credited pairs are at least required pairs. Its monthly cost is:

Self-hosted cost = B × H × G + O + E + F

The simple break-even volume, when the API price and workload shape are stable, is:

Break-even requests/month = self-hosted monthly cost ÷ (API cost per request)

where API cost per request is R × ceiling(D × C ÷ 100) × P under the example search-unit rule. This is an arithmetic threshold, not a recommendation. A production decision must also pass latency, reliability, data-governance, model-quality, and staffing gates.

vLLM’s scoring documentation shows what self-hosted implementation can look like: cross-encoder models accept two prompts, and its online serving supports score and Cohere-compatible rerank endpoints. The docs expose controls such as maximum query and document tokens, truncation, and the query/document inputs ^[4]. Those controls can change both pair throughput and quality, so record them beside your benchmark rather than treating a model name as a capacity estimate.

Worked cost example

Assume an English support-search application with these illustrative assumptions, not vendor quotes:

The $0.0025 API rate is a transparent scenario input, not a current universal price or a quote from a provider. Replace it with the production rate from your contract or invoice. The same warning applies to the $1.50 GPU rate and the $1,610 operating allowance.

Now test whether the two-GPU setup has enough capacity. Assume each GPU measures 800 pairs/second at the selected model, 50-document candidate set, batch policy, and target p95 SLO. Credit only 25% utilization to this workload because the service must absorb burstiness and preserve queue headroom:

800 pairs/s × 3,600 s/hour × 730 hours × 2 GPUs × 0.25 = 1,051,200,000 credited pairs/month

Required pairs are 12,000,000 × 50 × 1.0 × 1.03 = 618,000,000 pairs/month. Capacity therefore passes this illustrative screen. The self-hosted option is about $3,800/month, compared with $30,900/month for the assumed API meter, a difference of $27,100/month before tax and one-time migration work.

The corresponding arithmetic break-even is:

$3,800 ÷ ($0.0025 × 1.03) = approximately 1,476,000 requests/month

That result changes quickly. If average documents rise from 50 to 100, the API cost per request is still one search unit under this particular rule, but self-hosted pair demand doubles. If long documents create 1.8 chunks per document, the API may cross into two search units while self-hosted pair demand rises by 80%. If the measured p95-safe throughput is 250 rather than 800 pairs/second, two GPUs no longer provide the same capacity headroom. The point of the model is to expose those sensitivities instead of hiding them inside a single “API versus GPU” price.

For a first pass, enter your measured GPU rate, device count, utilization, throughput, and overhead in the Self-hosted GPU inference cost calculator. Then use the RAG, embedding and vector database cost calculator to keep vector retrieval, embedding, generation, and reranking as separate meters.

Throughput and SLO gates before switching

Cost is only comparable after the service passes these checks:

1. Same quality target. Run the API and self-hosted model on the same labeled queries, candidate set, language mix, and document formatting. A cheaper model that changes accepted-answer or citation quality is not an apples-to-apples substitute.
2. Same candidate and chunk policy. Record top-N retrieval, maximum document tokens, truncation side, chunk overlap, and any context compression. Cohere notes that query and document tokens share a per-document context limit and that overlong content can be chunked ^[3].
3. Same latency percentile. Measure p50 and p95, at minimum, under the concurrency and burst pattern that matters. Do not compare an API’s reported request latency with a self-hosted batch-one number when production uses a queue.
4. Capacity headroom. Size for peak requests, not only monthly averages. A GPU can look cheap at 70% average utilization while violating p95 during a five-minute burst.
5. Failure behavior. Include timeouts, retries, fallback ranking, health checks, standby capacity, and the cost of duplicate API attempts. A service with no fallback may have a lower spreadsheet cost and a higher incident cost.

The vLLM endpoint can help standardize the interface for a self-hosted cross-encoder, but the serving layer remains your responsibility ^[4]. NVIDIA’s guidance similarly treats containerized inference and scaling as part of production deployment, not as free capacity ^[5]. Put the recurring parts into O, E, and F rather than assigning them a zero price.

When the API is the better economic choice

Prefer an API when request volume is uncertain or seasonal, because you avoid paying for a warm GPU during idle periods. It is also often the better first choice when the team has no measured throughput benchmark, when the candidate set changes frequently, or when the required model is proprietary and would require a private deployment rather than a downloadable checkpoint.

An API can remain cheaper at high volume if its unit rate is low, if your traffic is bursty, or if self-hosting requires active-active redundancy and substantial on-call effort. Conversely, a fixed GPU can be attractive at steady high volume when the model is small, the candidate count is bounded, and the same accelerator can share cost with other inference stages. Treat shared-GPU savings as an allocation decision: document how much capacity the reranker actually consumes.

Use the Model pricing comparison page to normalize the API rate and unit definition. Once invoices arrive, use AI API invoice reconciliation to replace the estimate with actual search units, retries, credits, and billing-period totals.

Assumptions, exclusions, and limitations

This worksheet estimates recurring serving economics. It does not forecast traffic, guarantee quality, or select a provider. It excludes one-time migration, evaluation labeling, model fine-tuning, legal review, taxes, currency effects, reserved-capacity commitments, and the value of engineering time unless you add them to the inputs.

It also does not assume that a self-hosted open model matches a hosted model’s quality or that a reranker improves every retrieval set. If first-stage retrieval fails to surface the relevant document, reranking cannot recover it. Test candidate recall before spending on a faster reranker. Keep the search date beside every price, because provider meters, model versions, GPU availability, and private-deployment terms change.

FAQ

Is self-hosting a reranker always cheaper at high request volume?

No. High volume improves GPU utilization, but it does not guarantee a lower fully loaded cost. Candidate count, document chunking, p95-safe throughput, redundancy, network, observability, and operations can move the threshold. Recalculate with measured pairs/second and the actual API billing unit.

How many documents should I send to a reranker?

Send enough candidates to capture the relevant document without making the second-stage workload unnecessarily large. Start with a measured range, such as 20, 50, and 100 candidates, and compare recall, quality, p95 latency, and cost. Under Cohere’s documented pricing rule, up to 100 documents form one search unit, while long-document chunking can increase the counted documents ^[1]. Other providers may meter differently.

What throughput should I use for self-hosted reranker TCO?

Use sustained measured query-document pairs per second at the model, precision, document length, batch size, concurrency, and p95 SLO you intend to operate. Do not use a marketing maximum or a batch-one result if production batches requests. vLLM’s scoring API exposes query/document controls that can affect this measurement ^[4].

Does utilization reduce the GPU invoice?

Usually no. Utilization reduces the amount of useful capacity credited to paid GPU time. The instance or accelerator may still be billed for every running hour. Model idle time, burst headroom, and standby capacity in the numerator, then use utilization only in the capacity denominator.

Should I count a standby GPU as reranker throughput?

Count its cost, but count its throughput only when it serves the workload represented by your measured scenario. A hot standby protects availability while producing no pairs in normal operation. Add its recurring cost to redundancy or operations, and test failover capacity separately.

Can a reranker API reduce downstream LLM cost?

It can, but measure the effect rather than assuming it. A reranker may let the application pass fewer or more relevant chunks to generation. Cohere documents Context Focus as a way to return relevant passages within documents, which can change the downstream context sent to an LLM ^[3]. Compare total accepted-answer cost, not just the reranker line item.

What if my API price is per token instead of per search?

Replace the search-unit formula with the provider’s token meter. Keep query tokens, document tokens, chunking, retries, and any minimum billable unit explicit. The model structure still works: calculate monthly billable units, then compare them with self-hosted cost and capacity under the same quality and SLO constraints.