Research-backed calculator guide

Vector Quantization Savings vs Recall: Choosing Compression by Workload

Compare float32, int8, binary, and product-style vector compression with a transparent cost-versus-recall method for RAG and vector search workloads.

What “cost versus recall” should include

Recall is a quality measure, not a price. For a labeled query set, define recall@k as the share of the exact top-k neighbors that also appear in the approximate top-k results. If your product cares about whether a relevant passage appears anywhere in the context window, also measure hit rate@k and downstream answer quality. A change from 0.98 to 0.96 recall@10 may be immaterial for one RAG workload and unacceptable for another. A single average hides tail behavior, so inspect query slices such as long documents, rare terms, languages, tenants, and fresh data.

Cost has at least four layers:

1. Resident vector memory. This is the direct N × d × bytes calculation. It is usually the first reason to quantize.
2. Index overhead. HNSW links, IVF centroids, codebooks, document IDs, filters, metadata, allocator overhead, and replicas add bytes beyond the vector payload.
3. Rescoring and I/O. A compressed first pass may fetch more candidates and load full-precision vectors from disk for a second pass. That adds read volume, CPU scoring, cache pressure, and latency per query.
4. Lifecycle and operations. Training a product-quantization codebook, rebuilding an index, dual-writing during migration, monitoring recall, and retaining a full-precision shadow index all have cost.

AWS describes quantization as lossy compression that lowers memory and computation, and it gives a useful HNSW sizing equation that includes both vector dimensions and graph edges.^[1] Its example shows why vector payload alone is not a cluster-sizing answer: graph structure and search parameters can remain material after the vector representation is compressed. A research evaluation of RAG embedding storage similarly frames the decision as a performance-storage trade-off and evaluates several precisions against a float32 baseline rather than treating compression as universally safe.^[3]

The compression choices

Float32 is the reference point. It is easy to reason about and normally gives the highest fidelity among the choices in this comparison, but it can make RAM the dominant cost at scale. Keep it as a ground-truth index or as the vector representation used for rescoring even when the first-pass index is compressed.

Float16 or another 16-bit floating representation halves the vector payload in the simple byte calculation. It can be a useful low-risk test, but its quality depends on the model and implementation. It is not the same choice as int8: the bit layout, range, and rounding behavior differ.

Scalar int8 quantization maps each dimension to an 8-bit integer. The nominal vector payload is 4x smaller than float32. Calibration matters because ranges or buckets derived from an unrepresentative sample can distort the similarity ordering. The Sentence Transformers material hosted by Hugging Face describes both the 4x byte reduction and the importance of a sufficiently representative calibration set.^[2]

Binary quantization stores one bit per dimension in the idealized payload calculation, giving a nominal 32x reduction from float32. It is attractive when memory bandwidth and cache fit dominate, but it discards much more numeric precision. A common design is two-stage retrieval: use binary vectors to find a wider candidate set, then rescore those candidates with int8 or float32 vectors. Hugging Face reports a specific experiment in which binary retrieval with rescoring retained about 96% of baseline retrieval performance while reducing memory and disk use by 32x; that is evidence for the method, not a promise for your model.^[2]

Product quantization (PQ) splits a vector into subspaces and stores compact codebook identifiers. It can provide strong compression, but it introduces codebook training, approximate distance computation, and more parameters to tune. Do not compare “PQ” as if it were one fixed quality level. The number of subspaces, code size, training sample, distance method, and reranking policy all change the result. In some workloads, a simpler scalar method can deliver a better operational frontier even if its nominal compression is lower.

Worked cost example: a workload-first calculation

Assume one million vectors with 768 dimensions. The table below deliberately separates arithmetic from quality. The memory price is an illustrative scenario assumption of $0.10 per GB-month of provisioned vector memory, dated 2026-11-02. It is not a vendor quote and it excludes index overhead, replicas, storage, CPU, and network. The recall values are hypothetical acceptance-test outputs used to show how the comparison works; they are not benchmark claims.

The first row is not a production bill. It is a transparent baseline. If the deployment needs two replicas, multiply the resident-memory component by two. If the index consumes 1.4 times the vector payload after graph, IDs, and allocator overhead, multiply each row’s memory estimate by 1.4. If the service retains full-precision vectors for rescoring, add that store even when those vectors are not resident in RAM.

The useful comparison is not “binary is 32x cheaper.” It is whether binary plus its wider candidate pool and rescoring produces an acceptable cost per successful retrieval under the actual query rate. For example, with k = 10 and an oversampling factor of 4, the binary stage retrieves 40 candidates to rescore. At 1,000 queries per second, that is 40,000 candidate scores per second before filtering, and it may require 40,000 full-precision vector reads per second if the rescoring vectors are not cached. The right number is workload-dependent; measure it rather than treating the compression ratio as the whole TCO.

How to build the recall frontier

Use a held-out query set with exact nearest-neighbor results from the float32 baseline. For each candidate, record:

• recall@k and hit rate@k;
• p50 and p95 latency;
• resident RAM, on-disk bytes, and replica count;
• indexing time, calibration or codebook-training time, and rebuild frequency;
• candidate count and rescoring depth;
• CPU seconds, disk reads, and network bytes per query; and
• the application outcome, such as answer citation accuracy or task success.

Then calculate estimated monthly cost from the meters you actually pay. A simple model is:

monthly TCO = memory GB-month × memory rate + storage GB-month × storage rate + query rescoring CPU + disk-read cost + rebuild allocation + operational reserve

If you do not have a reliable cloud rate for a component, label it as an assumption and run a sensitivity range. For example, calculate the frontier at $0.05, $0.10, and $0.20 per GB-month rather than implying that one illustrative number is universal. Use the RAG, embedding and vector database cost calculator to keep embedding, retrieval, and generation assumptions in one model.

A candidate is dominated when another configuration is no more expensive, no slower, and no worse on the quality metric, with at least one strict improvement. Remove dominated candidates from the chart. The remaining points are the practical frontier. This prevents a team from arguing about a compression level that is both more expensive and less accurate than an available alternative.

When rescoring pays back

Rescoring is a quality recovery mechanism, not free accuracy. The first pass uses compressed vectors to retrieve a wider set. The second pass loads a higher-precision representation and recomputes similarity. More candidates can improve recall, but each candidate increases CPU work and may trigger random or cold-storage reads. If the full-precision vectors must remain in memory, the memory saving shrinks. If they stay on disk, the index may save RAM while adding I/O and latency.

The best way to size rescoring is to sweep the oversampling factor. Run factors such as 1x, 2x, 4x, and 8x while keeping the query set fixed. Stop increasing the factor when the recall gain is smaller than the added latency or query cost you can afford. Also test cache-warm and cache-cold conditions. A benchmark that reads the same tiny candidate set from cache can understate production I/O.

The AWS OpenSearch example describes this same two-phase pattern: retrieve candidates from a compressed representation, load full-precision vectors, and rescore before returning the final top-k.^[1] This is an implementation example, not proof that every vector database has identical costs.

Choosing by workload

Choose int8 first when the workload needs a substantial memory reduction, has a moderate quality floor, and can provide representative calibration data. It is a sensible baseline for a scalar-versus-product-quantization comparison because the storage arithmetic is simple and the operational path is usually less complex than a trained codebook.

Choose binary with rescoring when the corpus is large enough that RAM or memory bandwidth is the binding constraint, and the service can afford a candidate expansion plus higher-precision reads. It is a poor fit when every query must be served from a tiny latency budget with no cache or when missing a borderline neighbor is unacceptable.

Choose PQ when the index implementation is optimized for it, codebook training can be repeated safely, and the measured memory reduction justifies the training and tuning burden. Compare it against int8 using the same recall metric and query distribution. Product quantization is not automatically cheaper once codebook training, index rebuilds, and distance-table computation enter the operating model.

Keep float32 or a high-precision shadow copy when you need a trusted evaluation baseline, exact auditability, or reliable rescoring. The GPU inference TCO calculator is useful when the retrieval workload shares infrastructure with embedding or reranking inference; the vector index should not receive an artificially low cost allocation just because the GPU is already running.

Measurement checklist and limitations

Before changing production compression, freeze the embedding model, similarity metric, normalization, filters, top-k, and query sample. Generate exact neighbors on a representative slice. Compare quality by query segment, not only one mean. Test fresh and aged indexes because a codebook or calibration range can become stale as the corpus changes. Record whether the benchmark stores full-precision vectors for rescoring.

Do not turn published percentages into a guarantee. Hugging Face’s experiments show that some models retain high retrieval performance under binary or int8 compression, while other model and calibration choices can behave differently.^[2] The arXiv study of RAG embedding storage also reports that the best trade-off depends on the chosen quantization format, dimensionality reduction method, model, and evaluation setup.^[3] Its results are useful for method design, not a substitute for your own test set.

Finally, separate vector cost from the rest of the RAG bill. Embedding generation, metadata storage, filters, reranking, prompt tokens, LLM generation, observability, and replicas can outweigh vector bytes at modest corpus sizes. Use the model pricing comparison when the compressed index changes which reranker or generation model is affordable, and reconcile the estimate against actual usage with the AI API invoice reconciliation calculator. Quantization should improve the full pipeline, not merely one line item.

Frequently asked questions

Does 32x binary compression mean a 32x lower total bill?

No. It means the idealized vector payload uses one bit per dimension instead of 32 bits. Index graphs, IDs, metadata, replicas, full-precision rescoring vectors, disk reads, CPU, and operational work remain. Calculate the payload reduction first, then add those other components.

Is int8 always safer than product quantization for recall?

No. Int8 is often simpler to calibrate and explain, but recall depends on the embedding distribution and implementation. PQ can be better or worse depending on its subspace count, code size, training set, and search parameters. Measure both against the same exact-neighbor baseline.

How many candidates should binary search rescore?

There is no universal factor. Sweep a small set such as 1x, 2x, 4x, and 8x of k. Choose the smallest factor that meets the recall and downstream quality floor within the p95 latency and query-cost budget. Include disk-read and cache behavior in that test.

Should full-precision vectors be deleted after quantization?

Only if the measured quality, audit, and recovery requirements allow it. Keeping them on disk enables rescoring and makes it possible to compare the quantized index with an exact or higher-precision reference. Deleting them may reduce storage, but it removes that recovery path.

What recall metric should a RAG team use?

Use recall@k or hit rate@k for the retrieval layer, then measure a task-level metric such as citation correctness or answer success. If the application consumes a context window of 10 passages, recall@10 and hit rate@10 may be more informative than recall@1. Always define the metric before selecting compression.

Is a cheaper vector index always the better engineering choice?

No. A cheaper index can add latency, misses, retries, larger prompts, or more expensive reranking. Select a non-dominated point on the cost, quality, and latency frontier, and keep an explicit rollback path to the higher-precision configuration.