RAG Internals: How LLMs Retrieve and Cite Web Pages (2026) | AiVIS Cite Ledger Blogs

Key Takeaways

• RAG solves LLM hallucination and staleness by injecting retrieved context at inference time — training data is never enough for current, factual queries.
• Two separate pipelines exist: an offline indexing pipeline (crawl → chunk → embed → store) and an online query pipeline (retrieve → re-rank → prompt → generate).
• Query fan-out fires 3–8 sub-queries in parallel per user question, meaning your content must be retrievable across multiple phrasings of the same topic.
• Re-ranking with cross-encoders is a second-pass precision filter — raw vector similarity is not the final signal AI systems use to decide what to cite.
• Prompt drift (context window compression) means even retrieved content can be truncated before generation; short, extractable sentences survive; dense prose does not.

Article

# RAG Internals: How LLMs Retrieve and Cite Web Pages (2026)

Why RAG Exists: The Fundamental Limitation

Pure LLMs suffer from a hard set of constraints: a **knowledge cutoff** (training data ends at a fixed point), hallucinations on factual questions, inability to cite sources reliably, and staleness on current events or niche knowledge.

RAG solves this by treating the LLM as a *reasoner* that receives relevant external context at inference time. The seminal paper was "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Lewis et al. (Meta AI, 2020).

High-Level Architecture: Two Pipelines

Every production RAG system has two distinct pipelines:

**A. Indexing Pipeline (Offline / Ingestion)**

• Load documents (PDFs, HTML, Markdown, APIs, Notion, Confluence)
• Chunk them
• Embed the chunks
• Store in a vector database + metadata index

**B. Query Pipeline (Online / Inference-time)**

• Embed / rewrite the user query
• Retrieve relevant chunks
• Re-rank them
• Stuff into the LLM prompt
• Generate grounded response + citations

Indexing Pipeline Internals

Document Loading & Preprocessing

Modern systems use OCR + vision models or layout-aware parsers (Unstructured.io, LlamaParse, Docling) to handle tables, images (multimodal RAG), and structured data.

Chunking Strategies

Naive fixed-size chunking (512 tokens with 50 overlap) is common but suboptimal. Chunk size is a key hyperparameter: too small loses context; too large dilutes relevance.

Better approaches:

• **Recursive Character Text Splitter** (LangChain default) — respects sentence/paragraph boundaries
• **Semantic Chunking** — embed sentences and group when embedding similarity drops
• **LLM-based Chunking** — use an LLM to decide optimal chunk boundaries by meaning
• **RAPTOR-style Hierarchical Chunking** — recursive summaries at multiple levels (leaf to parent to grandparent)

Embedding Generation

Bi-encoders are most common: separate encoder for query and document, f

Enable JavaScript for the full interactive reading experience with related articles and discussion.

Cited external sources