Research-backed calculator guide

RAG Retrieval Capacity Planning for P95 Latency and Peak Concurrency

A practical RAG retrieval capacity planning method for turning peak QPS, concurrency, p95 latency, candidate depth, and replica headroom into a defensible capacity and cost estimate.

A RAG system can answer correctly and still fail its service-level objective when retrieval queues behind a traffic spike. RAG retrieval capacity planning is the process of sizing the vector-search path for the traffic shape and latency percentile you actually need, rather than dividing a monthly request count by a 30-day average.

The key separation is important. Retrieval usually includes query embedding, vector or hybrid search, metadata filtering, and sometimes reranking. Embedding ingestion, reranking inference, prompt construction, and final LLM generation have different resource curves. A capacity plan should show which part each number belongs to. AWS describes RAG as retrieving external knowledge before generation and converting data into vector representations for a vector database ^[1]. That architecture is why a single “RAG server” number can hide several independent bottlenecks.

What to measure before sizing

Peak QPS is not average QPS

Record the busiest sustained interval and the short burst separately. A dashboard showing 20 requests per second over a month does not prove that a 120-QPS campaign burst is safe. Use a defined window, such as the highest five-minute rate for a steady-state plan and the highest 30-second rate for burst protection. Also record whether QPS means user requests, retrieval calls, or fan-out searches. One user request that searches three namespaces may consume three retrieval operations.

Concurrency connects traffic to latency

For a stable slice of traffic, the approximate number of in-flight retrievals is:

concurrency ≈ arrival rate (QPS) × average retrieval latency (seconds)

This is the practical form of Little’s Law. Use average latency for the basic queue relationship, then test p95 and p99 to understand tail behavior. Do not substitute p95 latency into a claim about average concurrency without labeling the assumption. For admission control, reserve additional slots for slow queries, retries, and connections waiting on downstream services.

Percentiles are capacity signals

Track p50, p95, and p99 for the retrieval operation itself, not only the full answer. The p50 shows the typical path. The p95 is often the operational SLO. The p99 exposes filter combinations, cache misses, noisy neighbors, or queueing that averages conceal. Databricks lists latency as a production RAG challenge alongside retrieval quality, context-window limits, data freshness, and evaluation ^[2]. That combination matters: lowering latency by shrinking candidate depth may change recall, while increasing context can improve evidence coverage but slow later stages.

NVIDIA’s RAG performance guide separates concurrent requests, latency metrics, and requests-per-second throughput, and notes that retrieved context increases the model’s input sequence length ^[3]. Its illustrative figures are not a universal benchmark, but they reinforce the planning rule: query shape and chunk/context size belong in the workload specification.

Capture the query shape

At minimum, stratify tests by:

• vector dimension and index size;
• dense, sparse, hybrid, or keyword search;
• metadata-filter selectivity and the proportion of filtered requests;
• top_k returned by the vector search and the number retained by a reranker;
• warm-cache versus cold-cache behavior;
• read-only traffic versus concurrent upserts, deletes, or index refreshes; and
• single-query traffic versus fan-out and retries.

A benchmark that uses random vectors and no filters can be useful for a baseline, but it is not evidence for a production workload with selective filters and bursty tenants.

A provider-neutral capacity model

Define these variables before choosing a node type:

• Q_peak: peak retrieval requests per second.
• L_p95: measured p95 retrieval latency in seconds.
• C_peak: measured or estimated peak in-flight retrievals.
• q_replica: sustainable QPS per replica at the target p95, query mix, and index size.
• u_target: maximum planned utilization, for example 0.80.
• r_retry: reserve for retries and transient failures, for example 0.10.
• r_growth: near-term growth reserve, for example 0.20.

The first concurrency check is C_peak ≈ Q_peak × L_avg. If your measured peak has 90 QPS and 150 ms average retrieval latency, the baseline is 13.5 in-flight retrievals. Add a 25% burst and slow-query reserve and plan for about 17 active retrieval slots. This is a connection and queue check, not a replacement for throughput testing.

The throughput check is:

usable QPS per replica = q_replica × u_target
base replicas = ceil(Q_peak ÷ usable QPS per replica)
planned replicas = ceil(base replicas × (1 + r_retry + r_growth))

Use the reserve carefully. If q_replica was measured with failures and retries already included, do not add the same retry load twice. If the provider scales automatically, the result is still useful as a minimum target and alert threshold.

Worked example: four read replicas for a 90-QPS peak

Assume a support assistant has a peak of 90 retrieval QPS. A production-shaped load test finds that one read replica sustains 35 QPS while retrieval p95 remains below the team’s 180 ms target. The team chooses 80% planned utilization, 10% retry reserve, and 20% growth reserve. It uses two shards because the index and write/storage layout need two partitions; the shard count is not inferred from QPS alone.

The base result is four replicas because ceil(90 ÷ 28) = 4. The final reserve is a policy decision, not a mathematical guarantee. A team with reliable autoscaling and a short burst may keep four warm replicas and scale to six. A team with a strict no-queue launch window may provision five or six from the start.

The $350 node-month and $200 other-cost figures are explicit planning assumptions, not a vendor quote. For comparison, Pinecone’s documentation gives a provider-specific formula of node rate multiplied by shards and replicas, then adds storage, writes, and egress; it also shows an illustrative b1 example priced at $336.42 per month per node in one AWS region ^[4]. Actual rates, minimums, regions, discounts, and usage meters change. Treat the table’s $3,000 as a scenario for budgeting, not as current universal pricing.

The example excludes document embedding, query embedding if billed separately, reranking inference, LLM generation, application compute, observability, backups, and data transfer not included in the assumed $200. If the system calls a hosted reranker or generator, add those meters separately. If the vector service bills read units rather than fixed read nodes, replace the node formula with the provider’s read-unit model and retain the same measured QPS and percentile tests.

For a broader cost view, use the RAG, embedding and vector database cost calculator, then compare model-side assumptions with the model pricing comparison. If generation runs on your own GPUs, the self-hosted GPU inference cost calculator is a separate layer rather than a substitute for retrieval sizing.

Shards, replicas, and queue headroom

A shard answers “where can the index data live?” A replica or read worker answers “how much parallel read work can the system serve?” Some products combine those concepts in a node count, while others expose them independently. Do not increase shards merely because QPS rose if storage already fits; do not increase replicas alone if the index cannot fit or a filter requires more data locality.

Pinecone documents dedicated read hardware, warm data, and independent scaling of storage shards and read replicas ^[4]. That is useful as a mental model even when your provider uses different terminology. Its documentation also warns that performance depends on index size, vector dimensionality, metadata filtering, query patterns, throughput, and latency requirements. Those are the variables your own benchmark must hold constant.

Queueing is often the first visible symptom of insufficient headroom. Set an alert on queue wait time, not only on CPU. A retrieval p95 can look acceptable while a small number of requests wait behind a saturated worker pool. Track timeout rate, retry rate, rejected requests, and the age of the oldest queued request. A retry storm can turn a 90-QPS arrival rate into 110 or 130 retrieval attempts, so count attempts as well as user requests.

How to validate a retrieval SLA

Use a staged test rather than one maximum-throughput number:

1. Warm the index and record a baseline at low concurrency.
2. Replay a representative query mix at the expected steady peak.
3. Increase QPS in steps while recording p50, p95, p99, errors, queue time, and achieved QPS.
4. Repeat with the highest expected filter selectivity, candidate depth, and concurrent writes.
5. Run a burst test that exceeds the steady peak for the promised burst window.
6. Repeat after changing replicas, shards, cache policy, reranker, or region.

Pinecone’s scale-testing guide illustrates the shape of a reproducible test: 10 million records, 10 QPS, 10 concurrent users, a specified vector dimension and top_k, a latency target, and a reported percentile distribution ^[5]. The exact numbers are for its example workload. The transferable practice is to publish the dataset, request rate, concurrency, query parameters, duration, and percentile output so another engineer can reproduce the result.

Do not mix retrieval p95 with end-to-end p95. If retrieval takes 180 ms, reranking 70 ms, prompt assembly 20 ms, and generation time to first token 600 ms, the user-facing first-token path is not 180 ms. Keep a latency budget for each stage and identify whether the retrieval SLO is a hard boundary or only a diagnostic target.

Keep retrieval cost separate from the rest of RAG

A useful monthly model has at least six rows: vector storage, retrieval reads, writes and index refreshes, query embedding, reranking, and generation. Retrieval capacity planning answers the first three operational questions: how much data must be placed, how many reads arrive at peak, and how much provisioned or metered read capacity is needed. It does not price answer tokens or an embedding backfill.

This separation prevents a cheap retrieval estimate from hiding a costly generation path. It also makes optimization decisions more testable. Reducing top_k may lower retrieval work and context tokens, but it can reduce recall. Adding a reranker may improve relevance while adding a second latency and billing meter. Moving from on-demand reads to dedicated capacity may reduce tail variability while increasing idle cost at low utilization. Compare each change against both the p95 SLO and the quality metric.

If actual invoices do not match the estimate, use the AI API invoice reconciliation calculator to map usage rows back to retrieval, embedding, reranking, and generation assumptions. Keep provider pricing observations dated and record the region and billing unit beside the number.

FAQ

What is the difference between RAG retrieval capacity and total RAG capacity?

Retrieval capacity covers the work needed to find and return candidate context: query vectorization when it is part of the service, vector or hybrid search, metadata filtering, and any retrieval-side queueing. Total RAG capacity also includes document ingestion, embedding generation, reranking, prompt construction, application workers, and LLM generation. Size and measure those stages separately because they scale with different inputs.

Should I size from peak QPS or peak concurrency?

Use both. Peak QPS drives throughput and replica calculations. Peak concurrency exposes in-flight work and connection or queue pressure. Start with QPS × average latency as a consistency check, then validate concurrency directly in a load test. A burst can raise concurrency even when the five-minute average QPS looks modest.

What does p95 retrieval latency mean for an SLO?

It means that 95% of measured retrieval requests complete at or below the stated threshold during the defined window and workload. The remaining 5% can be slower, so track p99 and timeout rate as guardrails. State whether the measurement includes network time, query embedding, reranking, retries, and client-side queue wait.

How much headroom should a vector search cluster have?

There is no universal percentage. Choose a policy based on burst duration, autoscaling delay, retry behavior, and the cost of a miss. An 80% utilization target plus explicit retry and growth reserves is a transparent starting scenario, not a guarantee. Validate it against a burst test and revise it when traffic or index shape changes.

Do more shards always improve retrieval latency?

No. Shards can provide storage and distribution capacity, but more partitions can add coordination or network work and may not solve a saturated read executor. First identify whether the limit is storage fit, CPU, memory, disk, queueing, filtering, or network. Change one variable at a time and compare p50, p95, p99, achieved QPS, and quality.

When should I choose provisioned read capacity instead of on-demand?

Provisioned capacity is easier to justify when traffic is sustained and predictable enough to keep the hardware busy, and when a stable tail-latency target matters. On-demand capacity can be more economical for sporadic or highly variable workloads. Use your measured utilization and vendor billing model; do not infer break-even from a generic node price.