RAG Application Tutorial: Build a Retrieval-Augmented Generation Workflow for Trusted Answers

Overview

Retrieval-augmented generation, or RAG, combines two separate capabilities: search and generation. A retrieval layer finds relevant passages from a document collection, while a language model uses those passages to produce an answer. The model is not expected to remember every internal document. Instead, your application supplies the most relevant evidence at query time.

A typical RAG request follows this path:

1. A user submits a question.
2. The application converts the question into an embedding or search query.
3. A retrieval system finds related document chunks.
4. The application assembles those chunks into a controlled prompt.
5. The language model writes an answer based on the supplied context.
6. The interface displays citations or source references.

This architecture is useful for internal knowledge assistants, product documentation search, research tools, support workflows, and other AI applications that need access to changing information. It is not automatically the right choice for every problem. If the task depends mostly on behavior, formatting, or a stable transformation, a carefully designed prompt or structured tool call may be simpler. For a broader comparison, see how to choose between RAG, fine-tuning, and long-context prompting.

Step-by-step workflow

1. Define the answer contract

Begin with the behavior you expect from the application. Specify what questions it should answer, which sources it may use, what it should do when evidence is missing, and how citations should appear. A useful answer contract might require the assistant to answer only from retrieved passages, distinguish facts from assumptions, identify the source document, and say that it cannot find sufficient evidence when the context is incomplete.

Write several representative questions before building the pipeline. Include straightforward questions, questions requiring multiple documents, ambiguous questions, and questions that should produce a refusal or an uncertainty statement. These examples become your initial evaluation set.

2. Ingest and normalize documents

Collect the documents that the application is allowed to search. Common inputs include HTML pages, PDFs, markdown files, word-processing documents, tickets, and database records. During ingestion, extract the readable text and preserve metadata such as title, URL, section heading, author, document identifier, access scope, and last-modified value.

Normalization matters because retrieval quality depends on the text presented to the index. Remove navigation menus, repeated footers, broken layout artifacts, and irrelevant boilerplate where possible. Keep headings and list structure when they clarify meaning. A source record should remain traceable to the original document so that the final response can provide a useful citation rather than an opaque internal identifier.

3. Choose a chunking strategy

Chunking divides long documents into smaller passages that can be searched and placed into a model context. There is no universal chunk size that works for every collection. Start with logical boundaries such as headings, paragraphs, procedures, or question-and-answer pairs. If a section is too long, divide it further while preserving enough surrounding context to keep each passage understandable.

Small chunks can improve precision but may remove important context. Large chunks preserve context but may dilute the relevant passage and increase prompt size. Limited overlap between neighboring chunks can help when a sentence or definition crosses a boundary, but excessive overlap creates duplicate results. Store the original document and position information alongside every chunk.

4. Create embeddings and index the chunks

An embedding model converts each chunk into a numerical representation that captures semantic relationships. Store the resulting vectors in a vector index together with the text and metadata. The exact embedding provider can change over time, so keep the indexing process reproducible and record which model and preprocessing version created each index.

For many applications, semantic search works best when combined with lexical search. Semantic retrieval can find passages that use different wording from the question, while keyword matching can preserve exact names, identifiers, product codes, and technical terms. A hybrid approach is especially worth testing when users search for error messages or domain-specific vocabulary.

5. Retrieve and rerank evidence

At query time, convert the user question into a search request and retrieve more candidates than you plan to send to the language model. You can then filter by permissions, document type, date, or product area. If the first retrieval pass returns loosely related results, a reranking step can score the candidate passages against the full question and select a smaller, more relevant set.

Do not treat the highest similarity score as proof that a passage answers the question. Retrieval scores are signals for ranking, not truth values. Inspect whether the selected passages actually contain the required facts, and preserve enough metadata to explain why each passage was included.

6. Build a grounded generation prompt

Pass the retrieved passages to the model in a clearly labeled context block. Give the model a concise task, define the required response format, and state what to do when the context does not support an answer. A generic template can look like this:

System: Answer using only the supplied context. If the context is insufficient, say so. Do not invent citations.

Question:
{{user_question}}

Context:
{{retrieved_chunks}}

Response requirements:
- Answer directly.
- Separate supported facts from uncertainty.
- Cite the document title and section for each important claim.

Keep instructions separate from retrieved content. Treat documents as data, not as instructions that can override the application’s rules. This is an important defense against prompt injection inside stored documents. For structured answers, consider a schema or tool-based output method; compare the options in function calling, JSON mode, and tool use.

7. Return citations with the answer

Citations should be generated from trusted metadata rather than invented by the model. Give each retrieved chunk an internal reference, then map that reference to a document title, section, URL, or page location in application code. The user should be able to inspect the evidence behind an answer without seeing unnecessary internal fields.

Tools and handoffs

A maintainable RAG system separates responsibilities into stages. An ingestion job prepares documents. A chunking and indexing job creates searchable records. A retrieval service handles queries and access filters. A prompt assembly layer builds the model request. The response layer formats citations, logs outcomes, and applies safety or business rules.

Keep these stages loosely coupled. You should be able to replace an embedding model or adjust chunking without rewriting the user interface. Store configuration such as chunking rules, retrieval limits, prompt versions, and index versions with the application release. This makes it possible to compare changes rather than guessing why answer quality moved.

Access control belongs before generation. Filter documents according to the user’s permissions before their text enters the model prompt. Avoid assuming that hiding a citation is sufficient protection; once restricted content is included in context, it may influence the response. For privacy-focused development and testing, review how to build a local AI stack for private prompting and testing and how to build a document chatbot without leaking sensitive data.

Operational logs should capture the question, retrieved document identifiers, prompt version, model configuration, latency, and output status. Redact or minimize sensitive text according to your environment. Logs are most useful when they help you reproduce a poor answer without becoming an uncontrolled copy of the document collection.

Quality checks

Evaluate retrieval and generation separately. Retrieval checks ask whether the correct source appears among the returned candidates. Generation checks ask whether the answer is faithful to those sources, complete enough for the question, clear, and correctly cited. A fluent answer can still fail if retrieval selected the wrong passage or if the model added unsupported details.

Create a small, versioned test set containing questions, expected source documents, important answer points, and acceptable uncertainty behavior. Run it after changes to documents, chunking, embeddings, retrieval settings, prompts, or models. Include negative cases where the answer is absent from the collection. An application that confidently says “I do not have enough information” can be more useful than one that produces plausible but unsupported text.

Check common failure modes:

• Wrong chunk: The index retrieves a related passage but not the one containing the answer.
• Missing context: A definition, exception, or prerequisite was separated into another chunk.
• Overloaded context: Too many passages give the model competing or irrelevant information.
• Stale index: Updated documents are not reflected in searchable records.
• Permission leakage: Retrieval ignores user or document access boundaries.
• Unsupported completion: The model fills gaps with a likely-sounding answer.
• Weak citations: References are missing, ambiguous, or generated rather than mapped from metadata.

Human review remains valuable for high-impact workflows and for creating better test cases. See this guide to LLM prompt testing and evaluation workflows for a broader testing process. Track cost and latency alongside quality, since a technically accurate pipeline may still need changes to retrieval limits, caching, or model selection before it is practical to operate.

When to revisit

RAG is an updateable workflow, so revisit it whenever either the knowledge source or the model stack changes. Re-index after meaningful document updates, changes to extraction rules, or changes to chunking and embedding configuration. Re-run evaluation when you change the prompt, retrieval method, reranker, model, context limit, or citation format.

Review the system on a regular maintenance cycle even when no code has changed. Sample real questions, look for unanswered topics, inspect citations, and identify documents that are duplicated, outdated, inaccessible, or poorly formatted. Add representative failures to the test set instead of relying only on the original examples.

To put this tutorial into practice, start with one narrowly defined document collection and a small evaluation set. Implement ingestion, chunking, retrieval, grounded prompting, and citation mapping as separate steps. Test the pipeline with known questions before adding more sources. Then document the index version, prompt version, retrieval settings, and observed failure modes. That record gives your team a clear baseline for every future update and turns a one-off AI app tutorial into a maintainable LLM application development workflow.