What is Retrieval-Augmented Generation?

What is Retrieval-Augmented Generation?

Retrieval-Augmented Generation (RAG) is an artificial intelligence architecture that connects a large language model (LLM) to an external, searchable knowledge base. When a user submits a prompt, the system first retrieves relevant documents from an indexed database and injects those excerpts into the model's context window before it generates a response. This process allows the model to reference private, domain-specific, or real-time information without requiring retraining or fine-tuning.

Think of a standard LLM as a student taking a closed-book exam relying purely on memory. RAG turns that exam into an open-book test: the student receives the question, searches a library of reference manuals for the relevant pages, and writes an answer citing those specific passages.

How it works

A production RAG pipeline operates across two distinct phases: offline indexing and online query execution.

1. Ingestion and Indexing (Offline)

1. Document Loading and Parsing: Source data (PDFs, Markdown, databases, web pages) is collected and extracted into raw text structures.
2. Chunking: The text is split into smaller segments according to defined parameters, such as a chunk_size of 512 or 1000 tokens with a chunk_overlap of 50 to 200 tokens. Overlap preserves semantic continuity across chunk boundaries.
3. Embedding Generation: An embedding model converts each chunk into a high-dimensional dense vector representation (for example, vectors of 768 or 1536 dimensions).
4. Vector Storage: The resulting vectors and their corresponding text and metadata are stored in an approximate nearest neighbor (ANN) vector database.

2. Retrieval and Generation (Online)

1. Query Encoding: When a user submits a question, the same embedding model converts the input query into a dense vector.
2. Similarity Search: The vector database computes similarity scores (typically cosine similarity or dot product) against indexed chunks and retrieves the top_k candidates (often k = 3 to k = 10). Advanced systems frequently combine dense vector search with sparse keyword search (BM25) via hybrid search.
3. Re-Ranking (Optional): A cross-encoder re-ranking model re-scores the retrieved candidates to elevate the most relevant passages to the top of the list.
4. Prompt Augmentation: The system inserts the top retrieved chunks into a structured prompt template alongside the user query.
5. Inference: The LLM reads the augmented prompt and generates a grounded response, often appending source citations.

Retrieval-Augmented Generation vs Fine-Tuning

Retrieval-Augmented Generation and fine-tuning solve different problems in LLM system design. Fine-tuning adjusts the model's internal weights to alter tone, follow strict formatting rules, or master specific task behaviors, but updating its factual knowledge requires repeated retraining runs. In contrast, RAG leaves the base model weights untouched and supplies external non-parametric context dynamically at inference time. While fine-tuning teaches an LLM how to act, RAG provides the LLM with what to know.

A practical example

The following Python example uses LlamaIndex to build an in-memory vector index from text documents and execute a grounded RAG query:

from llama_index.core import Document, VectorStoreIndex

# 1. Prepare raw source documents
knowledge_base = [
    Document(text="Server Alpha deployment policy requires multi-region redundancy."),
    Document(text="Incident response runbook: P1 outages escalate to on-call within 5 minutes."),
]

# 2. Ingest, chunk, embed, and build vector index
index = VectorStoreIndex.from_documents(knowledge_base)

# 3. Create query engine with top-k retrieval configuration
query_engine = index.as_query_engine(similarity_top_k=2)

# 4. Execute retrieval and generation
response = query_engine.query("What is the escalation SLA for P1 outages?")

print(str(response))
print("Source nodes:", [node.get_text() for node in response.source_nodes])

What the output means

Running this script returns the grounded answer: "The escalation SLA for P1 outages requires escalating to on-call within 5 minutes." Inspecting response.source_nodes shows the exact chunk retrieved from the vector index, providing direct traceability for the generated output.

Tools and implementations

Tool	What it is for	Link
LlamaIndex	Data framework for building document ingestion, indexing, and query pipelines	developers.llamaindex.ai
LangChain	Orchestration framework for chaining retrievers, prompt templates, and LLMs	langchain.com
Qdrant	Open-source vector similarity search engine and vector database with payload filtering	qdrant.tech
Ragas	Evaluation suite providing automated metrics like faithfulness and context_precision	docs.ragas.io
Hugging Face Transformers	Reference implementations of neural retrievers (such as DPR) and generator models	huggingface.co

Where it goes wrong

RAG significantly reduces hallucinations, but it does not guarantee factuality:

• Retrieval Failures: If the retriever fetches irrelevant or noisy chunks (poor context_recall), the generator may hallucinate or fail to find the required answer.
• Out-of-Domain Embedding Drift: Standard embedding models may fail on specialized legal, medical, or code terminology unless adapted or paired with sparse keyword matching.
• Context Stuffing and Lost-in-the-Middle: Supplying too many chunks can exceed token budgets or cause the LLM to overlook critical facts buried in the middle of long prompts.
• Unfaithful Generation: An LLM can still contradict the retrieved text if its pre-trained parametric memory strongly overrules the context provided in the prompt.

Why it matters now

RAG has become the standard enterprise pattern for building factual AI assistants, internal search engines, and automated research tools because it allows organizations to update knowledge bases instantly without expensive GPU training runs. However, serving multi-step retrieval pipelines and injecting dense context into every prompt significantly increases infrastructure and token compute costs. In late October 2026, Google parent Alphabet saw search revenue rise 12% as rapid adoption of its compute-heavy AI Mode highlighted the monetization and cost pressures of running retrieval-augmented generation at global scale.

FAQs

How does chunk size affect RAG performance? Smaller chunks (e.g., 200–500 tokens) yield more precise vector matches but risk losing broader contextual meaning. Larger chunks (e.g., 1000+ tokens) preserve narrative context but dilute embedding specificity and consume more context window space.

What is hybrid search in RAG? Hybrid search combines dense vector retrieval (semantic similarity) with sparse keyword retrieval (such as BM25). A fusion algorithm, like Reciprocal Rank Fusion (RRF), merges the two result sets to catch both thematic meaning and exact keyword matches.

How do you evaluate a RAG pipeline? Frameworks like Ragas evaluate performance across component-specific metrics. Key metrics include context_precision and context_recall for the retrieval stage, alongside faithfulness and answer_relevancy for generation quality.

Can RAG work with structured data like SQL databases? Yes, but standard vector search is less effective for aggregations and relational queries. Structured RAG typically combines vector search with Text-to-SQL generation or knowledge graphs (GraphRAG) to query tabular information accurately.