Research-backed calculator guide

RAG Deduplication Cost: Remove Duplicate Chunks Before They Inflate Retrieval

Calculate the embedding, vector-storage, and retrieval-prompt cost of duplicate RAG chunks, then choose exact, near-duplicate, or retrieval-time deduplication without confusing chunk overlap with duplicate records.

unique_chunks = total_chunks × (1 − duplicate_rate)
embedding_savings = total_chunks × duplicate_rate × tokens_per_chunk × embedding_price_per_1M_tokens / 1,000,000
vector_bytes_saved = total_chunks × duplicate_rate × (id_bytes + metadata_bytes + 4 × dimensions)

That arithmetic only applies if the records really are duplicates. Sliding-window overlap creates shared text between neighboring chunks, but the chunks are usually not byte-identical. Treating every overlap region as a duplicate can remove context that a retriever needs. A safer policy is to hash exact canonical text before embedding, track source and version IDs, and use conservative near-duplicate review or retrieval-time diversity filtering when semantic similarity is high but not conclusive.

In the worked scenario below, removing 22% of 100,000 repeated chunks avoids an assumed $1.10 of embedding spend for one backfill and about 0.16 GB of vector-index bytes. The larger recurring saving appears only when those repeated chunks would have entered prompts: two redundant 500-token chunks across 10,000 queries represent 10 million avoidable prompt tokens. Those are assumptions for planning, not vendor quotes or a promise of savings.

What counts as a duplicate chunk?

Start with three different cases.

Exact duplicate record. The canonical extracted text and relevant metadata are identical, but the pipeline has created two records, perhaps under different IDs or after a repeated crawl. A cryptographic hash of normalized text is a strong first-pass key. Keep one record and preserve all meaningful provenance as references rather than storing multiple vectors.

Near-duplicate or superseded chunk. Two chunks share most of their meaning but one may contain a new date, exception, price, or policy version. Similarity alone cannot tell you whether to merge them. Keep both if they are materially different and route retrieval toward the current or authoritative version with metadata. Otherwise, designate a canonical record and mark the other as superseded.

Normal chunk overlap. A sliding-window splitter intentionally repeats a boundary region so that a fact spanning two windows is not cut in half. The two chunks can share 10–20% of their text while still containing different surrounding facts. That is not automatically a duplicate. The byte-exact RAG study describes this distinction explicitly: overlap across chunk edges is different from byte-identical chunks returned through multiple retrieval paths ^[1].

This distinction determines where cost belongs. Exact duplicates are usually cheapest to remove before embedding. Near-duplicates need more expensive comparisons and governance. Normal overlap is a chunking parameter to test against retrieval quality, not a deletion queue.

Where duplicate chunks create cost

The ingestion path has four cost surfaces. First, an embedding request consumes input tokens. Second, the vector store receives another write and stores the vector, ID, metadata, and possibly text. Third, later maintenance may pay for updates, deletes, re-embeddings, or imports. Fourth, the duplicate may be returned at query time, increasing egress, reranking work, prompt tokens, and generation prefill work.

The exact billing meters vary by vendor. Pinecone’s cost documentation separates read units, write units, storage, egress, embedding, and reranking; it also states that storage is based on total index size and that dense-vector bytes are calculated from dimensions at four bytes per dimension ^[3]. Its query example uses namespace size and a minimum read-unit charge, so deleting a small amount of data does not necessarily reduce every query’s bill immediately. In other words, calculate the bytes saved and the applicable plan threshold instead of multiplying duplicate count by an assumed query price.

The retrieval path has a different failure mode. Similarity search ranks relevance, not uniqueness. A top-k list can contain several records that say the same thing, leaving fewer slots for distinct evidence. The first byte-exact study reports that redundancy varies by corpus regime rather than being a fixed RAG constant ^[1]. A separate early paper tests a cross-attention filter and warns that exact matching can miss semantic redundancy while an aggressive similarity threshold can discard useful information ^[2]. Both findings support measuring your corpus rather than importing a generic duplicate percentage.

A cost model you can audit

Use the following variables for one ingestion run:

• N: chunks sent to ingestion.
• d: fraction that would be removed before embedding, from 0 to 1.
• t: average input tokens per chunk.
• e: embedding price in dollars per million input tokens.
• D: vector dimensions.
• m: average ID plus metadata bytes per record.
• s: your storage price per GB-month.
• q: monthly queries.
• r: redundant chunks that would otherwise enter a typical prompt.
• p: generation-model input price in dollars per million tokens.

Then:

removed_chunks = N × d
embedding_savings = removed_chunks × t × e / 1,000,000
index_bytes_saved = removed_chunks × (m + 4D)
monthly_storage_savings = index_bytes_saved / 1,000,000,000 × s
monthly_prompt_savings = q × r × t × p / 1,000,000

Use decimal gigabytes in this worksheet and state that assumption. If your provider bills gibibytes, substitute 1,073,741,824 bytes. Add write, delete, similarity-search, reranking, and cleanup-compute costs separately when the provider meters them. Do not treat a hypothetical price in this formula as a current vendor quote.

The break-even test is equally simple:

monthly_net_saving = monthly_prompt_savings + monthly_storage_savings − monthly_cleanup_cost
payback_months = one_time_cleanup_cost / monthly_net_saving

If the result is negative, do not deduplicate solely for a spreadsheet saving. You may still do it for correctness, freshness, access-control, or context-diversity reasons, but label that as a quality or governance decision rather than a claimed financial return.

Worked cost example

Assume an English support corpus has 100,000 chunks, each averaging 500 input tokens. A measurement sample suggests 22% are exact duplicates or safe, policy-approved near-duplicates. For planning, assume an embedding price of $0.10 per million tokens, an index with 1,536 dense dimensions, 1,008 bytes of ID and metadata per record, storage at $0.25 per decimal GB-month, and a generation input price of $0.50 per million tokens. These prices are illustrative assumptions, not vendor quotes.

The table shows why placement matters. If the 22,000 records are filtered before embedding, the $1.10 is a backfill saving. If they are already embedded and you only deduplicate at retrieval time, that embedding saving is gone, but the prompt saving can still exist. If the duplicates almost never rank together, the $5.00 is overstated; measure retrieved redundancy on representative queries. If the vector namespace remains below a provider’s minimum query meter, the 0.157 GB storage reduction may not reduce query units, even though it reduces storage bytes.

A cleanup job also has a cost. Suppose it costs $0.75 in compute and review time to classify the batch. Under these assumptions, the first month’s direct benefit is approximately $1.10 + $0.039 + $5.00 − $0.75 = $5.389. That is an illustrative payback calculation, not a forecast. Replace each input with invoice data, measured duplicate rate, and a query sample before approving a production change.

Choose the deduplication placement

Before embedding: Use canonical URL or source IDs, normalize only transformations that are semantically safe, hash extracted text, and reject exact repeats. This is the cheapest point for avoiding embedding and vector-write work. LangChain’s indexing API, for example, tracks documents with hashes and a record manager, exposes source IDs and cleanup modes, and warns that batching and cleanup choices affect redundant work ^[4]. Its behavior is a useful implementation reference, not a guarantee that every vector store will deduplicate your records.

During indexing: Use stable source IDs, chunk ordinals, parser version, embedding-model version, current/superseded flags, and a uniqueness rule. An update to a policy should supersede the old chunk rather than create two equally current records. Do not silently merge records with different permissions or provenance.

At retrieval time: Retrieve more candidates than the final context size, cluster or compare them, then select representatives based on authority, freshness, completeness, and access rules. This costs extra CPU or reranker calls, but it can protect a legacy index while you repair ingestion. Never deduplicate after retrieval by text alone if two sources have different authorization scopes.

In the prompt: A byte-exact set filter is the lowest-risk last gate for duplicate strings. Keep source labels and provenance. A semantic filter needs thresholds, tests for version conflicts, and an escape hatch for high-value exceptions. The research literature describes early semantic filters as dataset-dependent, so validate recall and answer correctness before deleting information ^[2].

Measure payback without fooling yourself

Track four rates separately: duplicate rate at ingestion, duplicate rate among retrieved top-k results, duplicate token share in assembled prompts, and answer-quality change after filtering. A corpus can have many duplicates that never rank together, or a small duplicate population that appears in nearly every high-volume query. These cases have very different economics.

Run a holdout evaluation with exact duplicates, near-duplicates, version pairs, and ordinary overlap. Compare unique evidence count, recall, citation correctness, prompt tokens, latency, embedding spend, vector bytes, and cleanup cost. Keep the original index or a reversible snapshot until the test passes. If a near-duplicate contains a newer exception, deleting it can reduce cost while making answers wrong.

Use the RAG, embedding and vector database cost calculator to model total retrieval and embedding spend, then connect the output to your observed duplicate rate. For a larger infrastructure view, compare the Self-hosted GPU inference cost calculator, because prompt reduction can change prefill demand even when the vector database bill is unchanged. When actual bills diverge from the model, use the AI API invoice reconciliation calculator to compare estimated token meters with provider usage.

FAQ

Does chunk overlap count as duplicate data?

Usually not. Overlap is an intentional shared boundary between different chunks. Remove it only when the entire normalized chunk is repeated or when an evaluation proves that the shared content is redundant for your task. A safe implementation distinguishes whole-string hashes from overlap percentage.

Is it cheaper to deduplicate documents before embedding?

Usually, yes, when the duplicate decision is reliable. Pre-embedding filtering avoids embedding tokens and vector writes as well as future storage. It does not automatically save retrieval cost, because query-unit billing and prompt savings depend on the provider and on whether the repeated records would have been retrieved.

Should I use cosine similarity to remove near-duplicates?

Use similarity as a candidate-generation signal, not as an unconditional delete rule. Related documents, translations, summaries, and old/new versions can have high cosine similarity while carrying different facts. Set a conservative threshold, preserve provenance, and send uncertain pairs to review or retrieval-time selection.

Can retrieval-time deduplication fix an already inflated index?

It can reduce repeated context and protect answer quality without immediately rebuilding the index. It does not recover past embedding spend, and it may add comparison or reranking cost. Pair it with an ingestion repair plan using stable IDs, hashes, versions, and reversible cleanup.

How do I know whether deduplication pays back?

Measure the duplicate rate, average chunk tokens, provider embedding price, bytes per vector record, storage rate, query volume, redundant chunks per prompt, generation input price, and cleanup cost. Plug those values into the formulas above, then run a quality holdout. A negative direct saving can still be acceptable for freshness or correctness, but it should not be presented as financial payback.