RAG Pipeline Cost Calculator
Retrieval-Augmented Generation looks free until your bill arrives and you realize you are paying to re-send thousands of context tokens to an LLM on every single query. The embedding step is a rounding error. The vector database is barely a line item. The real money disappears into generation — and it scales linearly with traffic. This calculator breaks a RAG pipeline into its four cost layers so you can see exactly where the spend lives, how much each retrieved chunk costs you per month, and whether a cheaper embedding model or fewer context chunks would move the needle more than switching LLMs.
Model Your RAG Pipeline
The RAG Cost Formula
A RAG pipeline bills you in four layers, and only one of them scales with traffic in a way that hurts:
Monthly = Embedding(re-index?) + LLM[(Q × (qTok + chunks×chunkSize))/1M × inP + (Q × outTok)/1M × outP] + Rerank(Q × chunks×chunkSize /1M × $0.10) + VectorDB(vectors/1M×store + Q/1M×read)
The embedding term is a one-time charge at index time (it recurs only if you re-index). The generation term is the monster: it multiplies your query volume by your context window. Because context tokens are input tokens, and input tokens are billed on every call, the retrieved-chunk count is the dial that matters most.
Why Chunk Size Is the Hidden Cost Dial
Every chunk you retrieve is multiplied by your query count and sent to the LLM as paid input. A 500-token chunk retrieved 5 times per query at 50,000 queries/month is 125M billed tokens. Drop to 300-token chunks or 3 chunks and that falls to 45M — a 64% cut in the dominant cost, with often negligible quality loss if your chunks are well-formed. Teams obsess over which LLM to use (a 2–4x input-price difference) while ignoring that chunk strategy can swing generation cost by 2–5x with zero model change.
- Smaller chunks — cheaper per query, but you may need more of them to capture the answer, and retrieval noise rises.
- Larger chunks — fewer retrievals, but each query bills more tokens and may hit context limits.
- Fewer chunks — the highest-leverage cost cut; pair with reranking to protect quality.
Worked Example: A 10,000-Document Support Bot
Index 10,000 documents of 4,000 characters each with 500-token chunks: that is ~10M source tokens → ~80,000 chunks → ~40M embedding tokens, costing about $0.80 once with text-embedding-3-small. Now serve 50,000 queries/month, each retrieving 5 chunks (2,500 context tokens) plus a 20-token question, answered with 400 output tokens on GPT-4o-mini. Generation bills 126M input tokens ($18.90) and 20M output tokens ($12.00) = $30.90/month. Reranking adds ~$12.50, Pinecone storage is negligible. Total recurring is about $43.40/month, with generation at 71% of spend. Switch to Gemini Flash and generation drops to ~$5/month; cut to 3 chunks and it drops another 40%. The embedding and vector DB costs never move the needle — generation and context do.
Frequently Asked Questions
What are the main cost components of RAG?
Four layers: embedding (index time), vector DB storage/reads, LLM generation (70–90% of monthly spend, driven by retrieved context), and optional reranking. Generation scales linearly with query volume.
Why does retrieved context dominate RAG cost?
Every query resends the retrieved chunks as input tokens. 5×500-token chunks at 50k queries = 125M billed input tokens — usually far more than output. Smaller chunks or fewer retrieved chunks is the biggest lever.
Is fine-tuning or RAG cheaper?
Volatile knowledge favors RAG (near-zero training, recurring generation). Static knowledge at very high volume can favor fine-tuning plus light RAG, since training front-loads cost but cuts per-query context.
How much does embedding cost?
Cheap: text-embedding-3-small is $0.02/M tokens. 10k docs → ~40M tokens → ~$0.80 once. Re-indexing makes it recur, but it is never the budget problem.
Does the vector DB add meaningful cost?
Usually not. pgvector/Weaviate self-hosted ≈ $0; Pinecone is ~$0.096/1M vectors-month plus tiny per-read fees. Storage is negligible next to generation.
Does reranking justify its cost?
At ~$0.10/M tokens it is modest ($12.50/month in the example). It pays off when better ranking lets you retrieve fewer chunks, cutting the larger generation bill.