Building a production-ready retrieval-augmented generation (RAG) application requires more than connecting a language model to a vector database. This guide explains the complete AI development workflow—from document ingestion and chunking to retrieval, grounded generation, evaluation, monitoring, and cost estimation—so you can design a RAG system that is measurable, maintainable, and easier to improve over time.
Overview
RAG combines a language model with an external collection of documents. When a user asks a question, the application retrieves relevant passages and places them in the model's context before generating an answer. The model does not need to store every piece of domain knowledge in its parameters; instead, the application supplies relevant information at request time.
A typical production RAG architecture contains these components:
- Ingestion: Collect files, web pages, database records, or other approved sources.
- Parsing and cleaning: Convert source material into usable text while preserving headings, tables, page references, and metadata where possible.
- Chunking: Divide documents into retrieval units that are large enough to preserve meaning but small enough to search precisely.
- Embedding: Convert each chunk into a numerical representation for semantic search.
- Indexing: Store embeddings, text, and metadata in a vector database or another searchable index.
- Retrieval: Find candidate chunks for a user query using semantic search, keyword search, metadata filters, or a combination.
- Generation: Give selected evidence to the language model with instructions to answer from that evidence.
- Evaluation and monitoring: Measure retrieval quality, answer quality, latency, failures, and cost.
RAG is useful when information changes regularly, must remain separate from model training, or needs to be traceable to source documents. It is not automatically the best solution for every AI feature. Compare it with long-context prompting and fine-tuning before committing to an architecture; the right choice depends on data size, update frequency, response requirements, and operational constraints. For a broader comparison, see how to choose between RAG, fine-tuning, and long-context prompting.
How to estimate
Estimate a RAG application in separate phases rather than treating it as one undifferentiated AI cost. This makes assumptions visible and helps you identify which design decision is driving the total.
1. Estimate ingestion and indexing work
Start with the amount of source material and how often it changes. A simple planning model is:
Indexed chunks = total source tokens ÷ average tokens per chunk
For operational planning, also estimate the number of documents, parsing time, embedding requests, storage requirements, and re-indexing frequency. A collection that changes once a year has different indexing needs from a knowledge base that changes every day.
2. Estimate query volume
Separate normal traffic from peak traffic. Record expected queries per day, average retrieved chunks per query, average input tokens, average output tokens, and the percentage of requests that require retries or follow-up tool calls.
A useful request-volume formula is:
Monthly requests = daily requests × active days × retry multiplier
The retry multiplier should reflect observed behavior after testing. Do not assume every request succeeds on the first attempt if your workflow includes query rewriting, reranking, safety checks, or structured output repair.
3. Estimate model and infrastructure usage
Separate recurring infrastructure from usage-based model operations. The model portion can be represented as:
Generation usage = requests × (average input tokens × input rate + average output tokens × output rate)
Use the actual rates for the model and deployment option you select. This article does not provide current vendor pricing because those inputs change. Include embedding generation, reranking, vector storage, database operations, application hosting, logging, monitoring, and network transfer in the broader estimate.
4. Estimate quality and operational outcomes
Cost alone is not an adequate decision metric. Track whether the system retrieves the right evidence and whether the final answer uses it correctly. Create a representative evaluation set containing questions, expected source documents, acceptable answer characteristics, and known unanswerable questions.
Useful measurements include retrieval recall, citation coverage, groundedness, answer correctness, refusal quality, latency, and failure rate. For a repeatable testing process, use the prompt engineering evaluation guide and the LLM prompt testing workflow.
Inputs and assumptions
Write down these inputs before selecting a vector database, embedding model, or generation model. The estimate is only as useful as the assumptions behind it.
- Source volume: Document count, approximate text size, file types, and proportion of duplicate or low-quality content.
- Update rate: How often documents are added, edited, deleted, or replaced.
- Chunking policy: Chunk size, overlap, whether headings are retained, and whether chunks are created by tokens, characters, paragraphs, or semantic boundaries.
- Metadata needs: Access permissions, department, date, document type, language, version, and source location.
- Retrieval strategy: Vector search, keyword search, hybrid search, metadata filtering, query expansion, and optional reranking.
- Context policy: Number of chunks supplied to the model, maximum context budget, and rules for removing redundant passages.
- Answer policy: Whether the model must cite sources, refuse when evidence is insufficient, quote text, or return structured JSON.
- Traffic pattern: Average and peak requests, concurrency, expected retries, and service-level latency targets.
- Data controls: Retention, access boundaries, redaction, audit logs, and handling of sensitive documents.
Chunking deserves special attention. Chunks that are too small may lose the context needed to answer a question. Chunks that are too large may make retrieval less precise and increase generation input. Begin with a defensible baseline, then compare alternatives using the same evaluation set rather than choosing a setting by intuition.
Metadata filters should be applied before or alongside semantic retrieval when users are only allowed to see a subset of the collection. Retrieval is not an authorization system by itself. Enforce permissions in the application and index design, then test cross-tenant and cross-role queries explicitly. For related privacy considerations, see how to build a document chatbot without leaking sensitive data.
Worked examples
Example 1: Small internal knowledge base
Assume a team has 2,400 documents averaging 1,500 tokens each. The estimated source volume is:
2,400 × 1,500 = 3,600,000 source tokens
If the initial chunking policy averages 600 tokens per chunk, the first approximation is:
3,600,000 ÷ 600 = 6,000 chunks
This is a planning estimate, not an exact count. Overlap, tables, repeated headers, parsing artifacts, and short documents will change the final total. If the application receives 300 questions per day over 30 days, the base monthly query volume is 9,000 requests. Add a retry multiplier only after observing the prototype. For example, a planning assumption of 1.08 would represent an estimated 9,720 total model-handled requests.
Next, multiply requests by the average retrieved context and output sizes, then apply the current rates for the selected model. Keep embedding and generation calculations separate because the indexing workload may be infrequent while query generation is continuous.
Example 2: Frequently updated support content
Assume a support collection contains 20,000 documents, but only 2 percent change on a typical day. A full re-index may be unnecessary. Estimate the daily update workload as:
20,000 × 0.02 = 400 changed documents per day
Design the ingestion pipeline to identify additions, edits, and deletions using stable document identifiers and content hashes. Re-embed only changed chunks, remove obsolete vectors, and retain version metadata for troubleshooting. If updates frequently alter headings or section boundaries, measure whether incremental chunking produces consistent retrieval results.
For both examples, compare a baseline retriever with a hybrid or reranked approach using the same questions. If a more expensive retrieval path improves answer quality only marginally, reserve it for difficult queries instead of applying it to every request. Likewise, use a smaller generation model for simple, well-grounded answers when evaluation shows that quality remains acceptable.
When to recalculate
Recalculate the estimate whenever a major input changes, not only when a provider changes its pricing. Review it after:
- Changing the embedding or generation model.
- Increasing chunk size, overlap, retrieved context, or output limits.
- Adding query rewriting, reranking, classification, or tool calls.
- Moving from a prototype collection to production-scale traffic.
- Changing document formats, update frequency, or retention requirements.
- Observing higher retry rates, latency, token usage, or storage growth.
- Changing database, hosting, observability, or security controls.
- Receiving new evaluation results that show weak retrieval or unsupported answers.
Use this practical review loop:
- Export a sample of recent queries, retrieved chunks, answers, citations, latency, and usage.
- Compare production measurements with the original assumptions.
- Identify whether the problem is ingestion, chunking, retrieval, prompting, model behavior, or infrastructure.
- Change one major variable at a time and rerun the evaluation set.
- Update the cost and quality estimate before rolling the change out broadly.
A production-ready RAG application is not defined by a particular model or database. It is defined by a traceable data pipeline, controlled access, grounded responses, measurable quality, and an estimate that reflects real usage. Keep the architecture modular, record the assumptions behind every optimization, and revisit the numbers whenever traffic, documents, models, or retrieval behavior changes.
For a final implementation review, use the prompt engineering checklist before shipping an AI feature. If you need to reduce usage without weakening tested outcomes, apply the principles in how to reduce LLM costs without hurting output quality.
