Estimating RAG Costs
AI pricing guide · updated 2026-08-07
Retrieval systems have a distinctive cost shape: a lot of tokens in, a short grounded answer out. That makes the input rate the number to watch, and it makes two levers, retrieval discipline and caching, worth more than any model swap.
Where the money goes
A retrieval request sends a system prompt, several retrieved passages and a question, then receives a comparatively short answer. Ratios of 80 to 20 in favour of input are common, and steeper is easy to reach.
Embedding costs are usually the smaller line. The 18 embedding models we track start at $0.02 per million input tokens, and embedding is a one-off per document plus a trivial cost per query. Generation, repeated on every request with a full context attached, is where the bill accumulates.
That is why the cheapest model for retrieval is often not the cheapest model overall: a low input rate matters more here than a low output rate.
Working the arithmetic
Take the tokens in a typical request: system prompt, plus retrieved passages, plus the question. Multiply by the input rate. Add the expected answer length multiplied by the output rate. Multiply by requests per day.
A concrete case: 400 tokens of instructions, four retrieved passages of 500 tokens each, a 60-token question and a 200-token answer. That is 2,460 input tokens and 200 output. On a model at $1.00 and $5.00 per million, roughly $0.0035 per request, or about $105 a month at 1,000 requests a day.
Then add the one-off embedding cost for the corpus and a small recurring cost for new documents and each incoming query.
The two levers that matter
Retrieve less. Returning four strong passages instead of twelve mediocre ones cuts input tokens by two thirds and frequently improves the answer, because models lose track of detail buried in long context. Tuning retrieval quality is a cost lever and an accuracy lever at once.
Cache the stable prefix. If your system prompt and instructions repeat across requests, prompt caching charges about a tenth for that portion. On a long fixed preamble this is the single largest saving available, and it applies to the part of the request you cannot shorten.
What estimates usually miss
Re-embedding. Changing embedding model means re-embedding the whole corpus, which is a real one-off cost worth knowing before you switch.
Reranking. A reranker adds a step and a charge, usually justified by better passage selection that reduces generation tokens, but it belongs in the model.
Retries and failed generations, which bill normally, and evaluation runs over your test set, which are easy to forget and simple to move to a batch endpoint at half price.
Frequently asked questions
What drives the cost of a RAG system?
Input tokens, by a wide margin. Retrieval sends a system prompt plus retrieved passages on every request and receives a short answer, so the input rate dominates. Embedding is usually a much smaller line: a one-off per document plus a trivial cost per query.
How do I reduce RAG costs?
Retrieve fewer, better passages, which cuts tokens and often improves accuracy at the same time. Then cache the stable prefix, since prompt caching charges roughly a tenth for repeated content and system instructions are exactly that. Choosing a model with a low input rate matters more here than its output rate.
How much does it cost to embed a document corpus?
Total the tokens in the corpus and multiply by the embedding model's input rate. The cheapest of the 18 embedding models we track starts at $0.02 per million tokens, so even large corpora are inexpensive to embed once. The recurring cost is only new or changed documents plus each incoming query.
See it in the data
Related guides
Terms used in this guide
Published by Tokenando. Last updated 2026-08-07. Figures in this guide are computed from our own pricing index and dated where they can move; see the methodology and corrections policy.