What Is RAG in AI? Architecture, Retrieval Pipelines, and Real Failures
A technical deep dive into retrieval-augmented generation: chunking mechanics, dense and sparse retrieval, cross-encoder reranking, and where production pipelines break.
Retrieval-Augmented Generation (RAG) is an architectural pattern that supplements a large language model with factual context retrieved dynamically from an external knowledge store during inference. Instead of relying solely on parametric knowledge baked into model weights during pre-training, a RAG system runs incoming user queries against a non-parametric index—such as a vector database or hybrid search engine—to select relevant passages and inject them into the prompt. This decoupling allows engineers to update domain knowledge instantly without model retraining, significantly reduces factual hallucination, and provides verifiable citations for generated claims.
First formalized by Patrick Lewis and colleagues at Facebook AI Research in 2020, RAG has evolved from a research-grade sequence-to-sequence exploration into the standard enterprise architecture for grounding LLMs on proprietary data.
The Core Architecture: From Corpus to Grounded Generation
Production RAG pipelines are composed of two asynchronous phases: an offline data ingestion and indexing pipeline, and an online query-retrieval-generation loop.
Ingestion: Documents ➔ Parser ➔ Chunker ➔ Embedding Model ➔ Vector/Keyword Index
Query Loop: Query ➔ Hybrid Retrieval ➔ Cross-Encoder Reranker ➔ Prompt Assembly ➔ LLM
1. Document Parsing and Chunking Mechanics
Raw documents (PDFs, Markdown files, HTML pages, database dumps) cannot be embedded as monoliths. Embedding models have strict context limits—typically 512 to 8,192 tokens—and compressing an entire 50-page manual into a single vector flattens granular details into useless average representations.
Ingestion requires splitting text into discrete passages called chunks. The primary chunking approaches include:
- Fixed-Window Chunking with Overlap: Text is split strictly by token or character counts (e.g., 500 tokens with a 50-token overlap). While simple to implement, it frequently severs sentences, tables, and logical paragraphs.
- Recursive Character Chunking: The splitter attempts to break on large semantic boundaries first (double line breaks
\n\n), descending to single line breaks (\n), sentence punctuation (.), and spaces only when a segment exceeds target size. This preserves paragraph integrity. - Document-Structure-Aware Chunking: Parsers parse Abstract Syntax Trees (ASTs), Markdown headers (
#,##), or HTML DOM hierarchies. Metadata such as parent section headers, document titles, and breadcrumbs are prepended directly to the chunk text so isolated paragraphs retain contextual hierarchy. - Semantic Chunking: A moving window calculates the cosine distance between embedding representations of consecutive sentences. Chunk breaks are injected whenever the distance exceeds a statistical threshold (e.g., the 95th percentile of distances across the document), ensuring every chunk covers a single semantic topic.
2. Dense Embeddings and the Bi-Encoder Model
To enable semantic similarity search, text chunks are passed through an embedding model—an encoder that maps variable-length text to a continuous dense vector in d dimensions (e.g., 768 dimensions for BAAI/bge-base-en-v1.5, 1,536 for OpenAI text-embedding-3-small, or 3,072 for text-embedding-3-large).
These models operate as bi-encoders: they encode the document passage d and the user query q independently into separate vectors:
v_doc = Encoder(d)
v_query = Encoder(q)
Because v_doc is computed offline during ingestion, billions of vectors can be indexed in advance. At runtime, the similarity between the query vector and candidate document vectors is calculated using geometric distance metrics, most commonly cosine similarity:
Cosine Similarity = (v_query · v_doc) / (||v_query|| * ||v_doc||)
Or the inner dot product if vectors are normalized to unit length (||v|| = 1).
3. Approximate Nearest Neighbor (ANN) Indexing
Computing exact cosine similarity between a runtime query vector and millions of document vectors (flat search) requires O(N * d) computations, which causes latency spikes at scale. Modern vector engines (e.g., Milvus, Qdrant, Pinecone, or pgvector) utilize Approximate Nearest Neighbor (ANN) indexing:
- Hierarchical Navigable Small World (HNSW): Constructs a multi-layer geometric graph where upper layers have long-range links for fast exploration and bottom layers have dense, local clustering for high precision. It provides logarithmic
O(log N)search time at the cost of high RAM usage. - Inverted File with Product Quantization (IVF-PQ): Partitions vector space into Voronoi cells and quantizes vectors into compact byte codes. It drastically reduces memory footprints for 100M+ scale datasets, though with a minor penalty to retrieval recall.
4. Hybrid Search: Dense Meets Sparse
Dense vector search excels at high-level semantic generalization (matching "cardiac arrest" with "heart attack"). However, it routinely fails on exact alphanumeric queries: SKU codes, error stack traces, IP addresses, drug names, and exact person names.
Modern RAG systems combine dense semantic retrieval with sparse keyword search (such as BM25, which scores documents based on term frequency and inverted document frequency). The results from dense and sparse queries are merged using Reciprocal Rank Fusion (RRF):
RRF_Score(d) = Σ [ 1 / (k + rank_i(d)) ]
Where rank_i(d) is the 1-based rank of document d in retriever i, and k is a ranking constant (empirically set around 60). RRF ranks items that appear consistently near the top of both sparse and dense candidate pools above items that score high on only one.
5. Two-Stage Retrieval and Cross-Encoder Reranking
Bi-encoders compress an entire passage into a single vector, losing token-to-token interactions between the query and the text. To recover precision, high-performing RAG architectures deploy two-stage retrieval:
- Stage 1 (Retrieval): The hybrid search engine retrieves a broad set of candidates (e.g., top 50 to 100 chunks) within 15–30 ms.
- Stage 2 (Reranking): A cross-encoder model (such as Cohere Rerank v3 or
bge-reranker-large) evaluates the query and each candidate chunk simultaneously. The input sequence is fed to the transformer as[CLS] Query [SEP] Document Chunk. Full self-attention operates across all query tokens and document tokens simultaneously.
The cross-encoder outputs a calibrated relevance logit between 0 and 1. While cross-encoders are far too computationally expensive to run against an entire database of 500,000 chunks, scoring 50 pre-filtered candidates takes roughly 40–80 ms. The pipeline discards low-scoring items and sends only the top 3–5 reranked chunks to the LLM.
6. Context Assembly and Generation
The reranked chunks are formatted with metadata (document name, section header, date) into an augmented prompt:
You are a technical assistant. Use the following verified context passages to answer the question.
If the context does not contain sufficient facts to answer, explicitly state that you do not know.
Do not use outside knowledge or extrapolate.
--- CONTEXT ---
[Doc 1 | Source: architecture_spec.md]
...
[Doc 2 | Source: deployment_guide.md]
...
--- QUESTION ---
[User Query]
The LLM processes the injected context in its attention layers, outputting grounded natural language grounded directly in the supplied passages.
A Worked Example: Tracing a Production Query
Consider an internal engineering support bot at a payments company answering the query:
"What is the timeout threshold for internal service-to-service gRPC calls in the payment gateway?"
Here is how data flows through the pipeline:
Step 1: Chunking the Knowledge Base
The source document is an internal architecture handbook containing 3,600 tokens. Using recursive chunking (target size: 400 tokens, 40-token overlap), Chunk #14 is created:
[Chunk #14 | Source: /docs/network/grpc-specs.md]"All inter-service gRPC communication routed through the internal Envoy mesh enforces a default cluster timeout of 2,500 ms. However, the payment gateway dispatch service overrides this default to mitigate upstream gateway lag: during peak operation, outbound gRPC deadlines are pinned to 1,200 ms. If downstream services fail to acknowledge within 1,200 ms, the circuit breaker opens."
Step 2: Dense Embedding and Hybrid Indexing
During offline ingestion, Chunk #14 was transformed into a 1,536-dimensional vector via an embedding model and indexed in an HNSW vector index. Concurrently, its tokens were indexed into a BM25 inverted index.
Step 3: First-Stage Candidate Retrieval
At runtime, the user's query is vectorized (v_query) and evaluated against the index:
- Dense Search (Cosine Similarity): The top 20 candidate chunks are fetched. Chunk #14 achieves a cosine similarity score of
0.791. It lands at rank 4 because chunks discussing general "payment gateway network latency" and "Envoy timeouts" share dense vocabulary and score0.824and0.805. - Sparse Search (BM25): The tokens
timeout,threshold,gRPC, andpayment gatewaymatch Chunk #14 strongly, ranking it at position 2. - Reciprocal Rank Fusion: With
k = 60, the RRF score prioritizes Chunk #14 into the top 3 of a combined 50-chunk candidate list.
Step 4: Cross-Encoder Reranking
The top 50 candidates are passed to a cross-encoder (bge-reranker-large). The cross-encoder attends over the exact phrase relationships: "timeout threshold", "gRPC calls", and "payment gateway".
- A distractor chunk describing general Envoy timeouts scores
0.14. - A distractor chunk describing client-to-API HTTP gateway timeouts scores
0.28. - Chunk #14 scores
0.963and is elevated to rank #1.
Step 5: Grounded Inference
The pipeline discards all candidates with a reranker score < 0.70, retaining Chunk #14 along with one supporting chunk on circuit breaker policies. The prompt is assembled and dispatched to the LLM.
Generated Output:
"Internal service-to-service gRPC calls originating from the payment gateway enforce a timeout threshold of 1,200 ms. While standard Envoy mesh services use a 2,500 ms default, the payment gateway overrides this deadline to 1,200 ms to prevent upstream lag [Source: /docs/network/grpc-specs.md]."
Where RAG Systems Fail: The Failure Points
Engineering an operational RAG pipeline involves overcoming distinct failure modes spanning information retrieval and generative attention, as documented in empirical software engineering research (e.g., Barnett et al., 2024):
1. Chunk Boundary Clipping (Context Fragmentation)
If a critical fact spans a split boundary—such as an entity defined at the bottom of Chunk A and its governing condition described at the top of Chunk B—the semantic meaning of both chunks is corrupted. The bi-encoder fails to match either chunk cleanly, or the retriever fetches only Chunk A, omitting the qualifying condition and leading to confident, incorrect assertions.
Mitigation: Use parent-child retrieval (small chunks for vector matching, linked to larger parent documents passed to the prompt) or sentence-window retrieval (expanding retrieved sentences with their immediate neighbors).
2. The "Lost-in-the-Middle" Degradation
Empirical research by Liu et al. (2023) demonstrated that large language models retrieve and reason over facts placed at the extreme beginning or extreme end of their input context far more reliably than facts placed in the center. If a RAG pipeline dumps 20 chunks into a 16,000-token context, and the golden fact resides in Chunk #11, the model frequently ignores it and hallucinates an answer.
Mitigation: Strict reranking to limit context to 3–5 chunks, placing the highest-scoring cross-encoder passages closest to the final user prompt.
3. Parametric vs. Non-Parametric Conflict
LLMs carry strong factual priors learned during pre-training. When retrieved internal documentation directly contradicts those priors—for example, a proprietary internal library named after a public framework, or custom protocols—models will often synthesize a hybrid falsehood or ignore the retrieved passage in favor of their parametric weights.
Mitigation: Hard-prompt constraints with negative constraints: "Base your answer exclusively on the clear facts stated in the provided text. If a fact contradicts common industry conventions, follow the provided text unconditionally."
4. Semantic Similarity vs. Relevance Divergence
Vector search measures topical proximity, not question-answering suitability. For example, the query "Why did the Q3 migration fail?" will yield high cosine similarity to chunks titled "Q3 Migration Success Metrics and Objectives" because they share dense vocabulary. However, the explanation for the failure might reside in an incident postmortem titled "Outage Report: Redis Shard Exhaustion", which shares zero vocabulary with the query.
Mitigation: HyDE (Hypothetical Document Embeddings), where an LLM first generates a hypothetical answer to the query, and that generated text is embedded for retrieval; or query rewriting pipelines that expand user queries into specific domain-relevant search terms.
5. Ingestion-Time Document Destruction
Many RAG failures occur before retrieval ever runs. Standard PDF text strippers extract tables into disjointed, unaligned text lines, stripping column-row relationships entirely. A retriever matching a table row loses its header context, yielding unparseable junk.
Mitigation: Use document layout analysis models (such as layout-aware OCR or vision-language parsing) that convert tables into cleanly formatted Markdown tables or JSON structures before chunking.
Trade-Offs: When NOT to Use RAG
RAG is not a universal architecture for enterprise AI problems. Implementing, managing, and debugging vector databases, embedding drift, and retrieval pipelines introduces significant operational overhead.
| Architecture Alternative | When It Outperforms RAG | Primary Trade-Off vs. RAG |
|---|---|---|
| Pure Long-Context Prompting | When analyzing 1–5 entire documents (e.g., analyzing a single 80-page legal contract end-to-end). | Significantly higher inference token costs and latency per query; vulnerable to middle-context attention drops on massive corpora. |
| Fine-Tuning (SFT / LoRA) | When teaching the model new syntax, tone, structured output schemas, or domain-specific reasoning behavior. | Cannot easily update dynamic factual knowledge without re-training; high upfront GPU compute costs; risk of hallucinations without grounding. |
| Text-to-SQL / Relational DBs | When answering quantitative, aggregate, or structured queries ("What was the median transaction value in March?"). | Rigid schemas; cannot handle unstructured semantic text queries; requires precise schema mappings. |
| GraphRAG / Hierarchical Clusters | When performing global corpus summarization ("What are the top 5 complaints across all 40,000 support tickets?"). | Substantially higher indexing latency and cost; requires multi-hop graph generation and community summarization. |
Do Not Use Standard RAG If:
- Your questions require holistic corpus aggregation. Vector search is an extractive needle-in-a-haystack tool. Asking a standard RAG pipeline "What are the primary themes across our entire patent portfolio?" fails because it retrieves 5 isolated chunks out of 100,000, missing the forest for the trees.
- Your data is fundamentally tabular and numerical. Calculating averages, sums, or correlations across database rows via vector similarity matches arbitrary numbers rather than computing mathematical operations. Use Text-to-SQL or analytics engines instead.
- Your latency budget is under 150 ms. An end-to-end RAG request requires: query embedding generation (15–30 ms), vector/ANN search (10–25 ms), cross-encoder reranking (40–80 ms), and LLM generation (200–1,000+ ms). If hard SLAs demand sub-100 ms responses, pre-computed caching or fine-tuned edge models are required.
Common Engineering Traps
- Over-relying on cosine similarity thresholds. Setting an arbitrary cosine similarity cutoff (e.g.,
similarity > 0.82) to filter out irrelevant chunks is fragile across different embedding models and domains. In high-dimensional spaces, cosine distances concentrate within narrow bands. Use a cross-encoder's calibrated sigmoid score or relative gap filtering rather than raw cosine cutoffs. - Ignoring embedding model asymmetric training. Many state-of-the-art embedding models (such as
bge-large-en-v1.5ore5-large-v2) are trained asymmetrically. They require specific prompt prefixes for queries (e.g.,"Represent this sentence for searching relevant passages: ") while indexing document passages without the prefix. Omitting the query instruction significantly degrades retrieval recall. - Failing to implement chunk metadata filtering. Searching an entire enterprise corpus globally for every query wastes compute and invites distractor passages. Production architectures use metadata extraction at ingestion (e.g.,
tenant_id,project,document_type,created_at) and perform pre-filtered vector searches (WHERE tenant_id = 'org_42') to constrain the search space.
FAQs
How does RAG differ from model fine-tuning?
Fine-tuning updates the internal weights of a neural network to teach it specific styles, tones, dialects, or task-specific reasoning patterns. In contrast, RAG keeps the model weights frozen and supplies external facts dynamically inside the prompt context at runtime. RAG is best for dynamic, factual knowledge that changes frequently, while fine-tuning is best for teaching behavioral form and specialized syntax.
Why is a cross-encoder needed if I already have a vector database?
A vector database relies on bi-encoders, which compress queries and documents into isolated vectors independently to enable fast similarity search across millions of records. This independent compression discards fine-grained cross-token interactions. A cross-encoder processes the query and candidate chunk together through joint self-attention layers, yielding vastly superior ranking accuracy on the top candidates at the cost of higher latency.
What chunk size should I select when building a pipeline?
A practical default for mixed technical documentation is 400 to 512 tokens with a 10% to 15% overlap using recursive character splitting. If queries target specific factual values (such as SKU lookups or definitions), smaller chunks (128–256 tokens) yield higher retrieval precision. If queries demand broader procedural context, larger chunks (800–1,024 tokens) or parent-document retrieval methods should be employed.
How do you systematically evaluate a RAG pipeline?
Engineers evaluate RAG using the "RAG Triad" metrics pioneered by frameworks like Ragas and TruLens: Context Relevance (did the retriever return only passages relevant to the query?), Groundedness / Faithfulness (can every claim in the generated output be mathematically traced to the retrieved context?), and Answer Relevance (did the output directly address the user's core intent without extraneous drift?).
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020) arxiv.org
- Seven Failure Points When Engineering a Retrieval Augmented Generation System (Barnett et al., 2024) arxiv.org
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) arxiv.org
- Rerankers and Two-Stage Retrieval (Pinecone) pinecone.io
The stories that matter for people who build with AI, with the source for every claim. Free, no spam.
Feed, daily deep-dive and bytes — readable offline, with push alerts for the topics you follow.