Research-backed calculator guide

Retrieval Quality per Dollar Benchmark: A Reproducible RAG Workbook

Learn how to compare RAG retrieval quality per dollar with fixed data, recall, latency, usage meters, and a reproducible worked benchmark.

cost per accepted retrieval = total benchmark cost ÷ accepted retrieval outcomes

An accepted outcome is a query that meets your predeclared quality and service thresholds. For example, you might require recall@10 of at least 0.92, grounded-answer pass rate of at least 0.85, and p95 retrieval latency of no more than 600 ms. A cheaper system that misses the quality threshold is not a cheaper production choice; it is an ineligible configuration for that service level.

This approach also keeps one-time and recurring costs visible. Embedding a corpus, storing vectors, reading or fetching records, reranking candidates, and returning data can all belong in different meters. Pinecone documents separate read units, write units, storage, egress, embedding, and reranking meters rather than one universal “search price.” ^[1] Qdrant describes cloud billing in terms of resources such as vCPU, memory, storage, backups, and inference tokens, with the free tier limited to stated resources. ^[2] OpenAI’s embeddings documentation states that embedding requests are billed by input tokens and that vector dimensions affect storage and compute trade-offs. ^[3]

What the benchmark should measure

Start with a frozen evaluation package. It should contain a versioned corpus snapshot, document IDs, the exact chunking rule, a query set, relevance judgments, metadata filters, and a definition of a successful answer. Keep a holdout slice for final comparison. If you regenerate embeddings or change the corpus during a run, you are measuring multiple changes at once.

The query set should represent the work your system will serve. Include short fact lookups, multi-hop questions, ambiguous terms, filter-heavy queries, long questions, and queries with no supported answer if those occur in production. Label not only whether a relevant document exists, but which chunks are relevant and whether the evidence is sufficient for the requested answer. A retrieval-only score can reward a system that returns a relevant paragraph but still fails the application’s source or answer requirement.

Use retrieval metrics that match the decision. Recall@k measures whether the judged relevant material appears in the first k results. Precision@k penalizes irrelevant items in that cut. nDCG@k gives more weight to highly ranked relevant items when your labels have grades. Report the cutoff, relevance-label policy, and the exact ground-truth method. Do not compare recall@5 from one run with recall@20 from another and call the difference a quality gain.

Add a downstream acceptance measure. One practical definition is “the returned context contains enough approved evidence for a reviewer or deterministic checker to mark the answer supported.” If an LLM judge is used, version the judge prompt and model, sample human-reviewed calibration cases, and report the judge cost separately. A judge is an evaluation instrument, not ground truth by itself.

Latency belongs beside quality. Record at least p50 and p95 retrieval latency, and preferably p99 during the same traffic shape. Include query embedding, vector search, metadata filtering, reranking, and network time in named fields. NVIDIA’s retrieval sizing guidance distinguishes the retrieval pipeline from the generation path and describes concurrency, time-to-first-token, end-to-end latency, throughput, and performance per total-cost-of-ownership dollar as separate metrics. ^[4] That is a useful reminder that “fast” needs a boundary and a workload.

Freshness is another control. Measure the delay between a document write being acknowledged and that document becoming retrievable under the relevant filter. A benchmark that only runs against a static, warmed index can hide the cost of updates, compaction, replication, and delayed visibility. If production has an 80/20 read/write mix, use that mix or disclose why a read-only test is representative.

A reproducible workbook layout

A minimal workbook can use one row per system and one column per controlled input or reported result. Keep these fields in the run manifest:

1. Run identity: benchmark version, corpus hash, query-set hash, index version, date, region, and software versions.
2. Retrieval configuration: embedding model, dimensions, distance function, chunk size and overlap, top-k, reranker, index type, search parameters, filters, and namespace or collection.
3. Traffic shape: query rate, concurrency, read/write ratio, warm-up duration, measurement duration, batch size, and freshness test procedure.
4. Quality: recall@k, precision@k or nDCG@k, grounded-answer pass rate, unsupported-answer rate, and no-answer behavior.
5. Performance: p50, p95, and p99 retrieval latency, end-to-end latency if measured, throughput, error rate, and freshness lag.
6. Cost: embedding input tokens, embedding dollars, write units, read units, storage GB-hours or monthly allocation, egress bytes, rerank requests, compute time, and any minimum commitment allocation.

The cost denominator must be explicit. “Per query” is useful for a steady-state operating estimate, but “per accepted outcome” is more decision-relevant when systems have different failure rates. Keep one-time corpus indexing cost separate from recurring query cost unless the decision horizon says otherwise. For a monthly budget, allocate backfill cost over the chosen number of months and show the undiscounted figure as well.

Vendor meters are not interchangeable. Pinecone, for example, documents a minimum of 0.25 read units for a query and says query cost is based on namespace size rather than top-k, while writes and storage use different calculations. ^[1] That means a top-k-only spreadsheet can materially misstate a provider bill. Qdrant’s page says usage is billed hourly and includes compute, memory, storage, backups, and paid-model inference tokens, so a fair comparison needs an allocation rule for idle capacity and high-availability replicas. ^[2]

Embedding storage deserves its own check. OpenAI documents default vector lengths of 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large, while also documenting a dimensions parameter for shortening embeddings. ^[3] At four bytes per floating-point value, a raw 1,536-dimensional vector is 6,144 bytes before IDs, metadata, index overhead, replicas, or compression. A 768-dimensional vector is 3,072 raw bytes. These are arithmetic assumptions, not a provider storage quote. If you change dimensions, rerun quality and latency tests; storage savings alone do not establish equivalence.

Worked cost example

Assume a US English support corpus with 100,000 labeled queries. The benchmark has 100,000 query embeddings, a fixed top-k of 10, the same metadata filters, and a 30-minute warm-up followed by a two-hour measurement window. The production target is recall@10 at least 0.92, grounded-answer pass rate at least 0.85, and p95 retrieval latency no more than 600 ms. The figures below are deliberately illustrative planning assumptions, not vendor quotes.

The arithmetic for System A is $52 ÷ 88,000 = $0.0005909, rounded to $0.000591 per accepted outcome. Its accepted-outcome count is not simply the number of retrieved documents. It is 100,000 queries multiplied by the 0.88 grounded-answer pass rate. System B looks cheaper on raw run cost, but it is not a qualifying configuration because both quality measures are below target. System C has the best quality and nearly the lowest eligible cost, yet its 610 ms p95 misses the declared latency service level by 10 ms.

For a monthly estimate, suppose the same quality and cost shape repeats over 2 million queries, with no change in corpus size or traffic mix. System A’s planning estimate is 2,000,000 × 0.88 = 1,760,000 accepted outcomes. If the illustrative run cost scales linearly, 20 runs × $52 = $1,040, producing $1,040 ÷ 1,760,000 = $0.000591 again. That linear extrapolation is only an estimate. It excludes tier changes, minimum commitments, replica growth, cache effects, peak capacity, failures, index rebuilds, and human review.

A more honest workbook also reports a sensitivity range. If the grounded pass rate for System A falls from 0.88 to 0.80 under a new traffic slice, accepted outcomes fall to 80,000 and cost per accepted outcome rises to $0.000650. If reranking adds $8 to the run but increases grounded pass rate from 0.80 to 0.88, compare the extra $8 with the accepted-outcome gain rather than treating reranking as automatically good or bad. The result is workload-specific.

How to compare providers without creating a false precision

Normalize units before comparing dollars. Keep the raw provider meter in the workbook, then translate it into the same run horizon. For one provider that may mean read units plus storage and egress. For another it may mean hourly vCPU, RAM, disk, replicas, and backups. Include the assumptions that connect usage to money: region, currency, tax treatment, exchange-rate date if applicable, reserved or on-demand pricing, and whether free allowances are consumed by this workload.

Do not silently assign a zero to a meter you did not measure. Write “not measured” or “excluded” and show a separate subtotal. If a vendor publishes a current rate, record the retrieval date and unit. If you use an internal invoice or a planning rate, label it as an estimate. Pinecone’s documentation itself separates how usage is measured from the page where current rates are listed, so a workbook should not turn a usage-unit formula into a price quote without a rate source. ^[1]

When self-hosting, include operational capacity rather than only the server’s hourly sticker price. A vector service may need replicas, failover, backup storage, monitoring, network transfer, on-call coverage, and an allocation for idle capacity. For a GPU-backed reranker or embedding service, a companion self-hosted GPU inference cost calculator can help separate hardware utilization from model quality. For model and embedding rate changes, use the model pricing comparison and record the effective date. When the test result is later compared with invoices, reconcile the run ledger with the AI API invoice reconciliation calculator.

What this benchmark excludes

This workbook does not predict search rankings, traffic, revenue, indexing, or first-week impressions. It does not prove that a benchmark winner will remain the winner after a corpus change. It does not replace a security, privacy, residency, or reliability review. It also does not treat one public benchmark score as a universal measure of retrieval quality.

Exclude generation-token cost when the question is strictly vector retrieval, but state that exclusion. Include it when the accepted outcome is a complete grounded answer and generation is part of the comparison. Exclude migration and backfill costs only when all candidates share the same migration burden; otherwise model them as one-time costs. If a provider’s minimum monthly charge is larger than the measured usage, allocate the minimum to the test and show the marginal usage subtotal separately.

FAQ

Is recall per dollar the same as retrieval quality per dollar?

No. Recall per dollar is one useful slice: a recall metric divided by a chosen cost denominator. Retrieval quality per dollar should define the quality metric and the decision outcome first. A system can have high recall but poor ranking, excessive latency, weak freshness, or poor grounded-answer support. Use recall@k, nDCG@k, p95 latency, and accepted outcomes together when those dimensions matter.

Should I use nDCG or recall@k?

Use recall@k when the key question is whether all necessary evidence appears in the candidate set. Use nDCG@k when ranking position matters and you have graded relevance labels. Many teams report both. Keep k, label definitions, query weighting, and no-answer handling identical across candidates. Do not choose the metric after seeing which system wins.

How many queries are enough for a RAG cost benchmark?

There is no universal number. Use enough queries to cover important traffic slices and produce stable estimates for the metrics that drive your decision. Report the sample size, confidence interval or bootstrap range where practical, and the composition of the set. A small, carefully labeled set can be useful for an early regression check, but it should not be presented as a production-wide quality estimate.

Should indexing and embedding cost be included?

Include them when the decision horizon includes corpus creation, migration, or regular refreshes. For steady-state query economics, show indexing and embedding as separate one-time or refresh subtotals. If you amortize them, state the number of months or refresh cycles. OpenAI documents input-token billing for embeddings, while the vector store may charge separately for writes and storage, so one “retrieval price” can conceal several independent meters. ^{[1] [3]}

Can a cheaper vector database be declared the winner?

Only after the candidate meets the same quality, latency, freshness, reliability, and compliance gates. Compare marginal and fully allocated cost separately. A low measured bill may reflect a free tier, a small static index, read-only traffic, or omitted replicas. A higher bill may buy capacity or a service level your workload requires. The benchmark should make those assumptions visible rather than burying them in a single score.