RAG: from documents to answers [2]

Image: STScI-01KJR33K9RKGM84B2RK8AGPT8W — NASA/ESA Hubble Space Telescope

In the previous post we talked about context: what an agent sees, why it matters, and what happens when the context gets too large. The natural question is: what do you do when the knowledge you need doesn’t fit in context at all?

That’s what RAG solves.

What RAG is — and what it isn’t

RAG stands for Retrieval-Augmented Generation. The name is accurate: you retrieve relevant information, then use it to augment the prompt before generating a response.

What RAG is not is fine-tuning or training. You’re not baking knowledge into the model permanently. You’re injecting it at inference time, only when it’s needed. This makes it practical for real use cases: documents that change, knowledge bases that grow, information that needs to stay up to date.

The canonical scenario: you have 50 documents, each 50 pages long. Passing all of them into context on every request is technically impossible and economically absurd. RAG lets you retrieve only what’s relevant to the specific question being asked.

The pipeline

1. Parsing

Before you can index anything, you need to extract the content from your documents. Parsing means identifying the different elements in a file: text, tables, images. The output is typically a clean Markdown reconstruction of the document, preserving the original structure and order.

This step is less trivial than it sounds. PDFs are notoriously hostile to extraction. Tables break into garbage, multi-column layouts lose their reading order, images contain text that no parser sees. Getting this right is foundational — garbage in, garbage out.

2. Chunking

Once you have clean text, you split it into chunks — smaller units that can be indexed and retrieved independently.

Two parameters control this:

Chunk size: how many tokens per chunk
Overlap: how many tokens are shared between adjacent chunks, to avoid cutting a sentence or paragraph mid-thought

There’s no universal value for chunk size. Research benchmarks generally suggest 256–512 tokens as a starting point: a 2026 analysis of 7 chunking strategies on 50 academic papers found recursive 512-token splitting at 69% retrieval accuracy. For factual queries, smaller chunks (256 tokens) tend to work better; for analytical tasks requiring broader context, 1024+ tokens may be appropriate. A practical ceiling to keep in mind: quality tends to drop noticeably around 2,500 tokens per chunk.

Chunking isn’t only about length. You can also split by logical unit — paragraph, section, chapter — to preserve the integrity of a concept. Semantic chunking, which splits based on topic shifts rather than token count, outperforms fixed-size splitting on certain tasks but costs more to compute.

3. Embedding

Each chunk gets transformed into a vector by an embedding model. A vector is a sequence of numbers that captures the meaning, semantics, and context of the text. The more powerful the embedding model, the longer the vector — and the more nuanced the representation.

You can enrich each chunk’s metadata at this stage: which document it came from, which page, which section. This metadata matters when you need to show users a source reference — “this answer comes from document X, page 12.”

4. Vector database

The vectors get stored in a vector database. How you choose it depends on your context:

Serverless (local) — libraries like ChromaDB or LanceDB that run in-process, no server required. Good for development and for production scenarios where the knowledge base is relatively static and the load is low.

Server-based — a proper database instance. Necessary when multiple users can update the knowledge base concurrently, when you’re running on containers, or when you’re dealing with millions of vectors.

My preference based on what I’ve worked with: pgvector (PostgreSQL extension) hits the right balance of maturity, compatibility with standard tooling, and production reliability. With HNSW indexing it handles millions of vectors at low-millisecond latency. ChromaDB is excellent for getting started fast. Pinecone or Weaviate make sense only at very large scale (100M+ vectors).

5. Retrieval

When a user asks a question, the question itself gets embedded using the same model, and the database returns the most similar chunks via a similarity search.

The main similarity metrics:

Cosine similarity — measures the cosine of the angle between two vectors; the closer to 1, the more similar. The standard choice for semantic search
Keyword search (BM25) — exact or fuzzy term matching; fast, and useful when terminology is specific and consistent
Hybrid search — combines both, with configurable weights. The most robust option in practice: you get semantic understanding and keyword precision

For how many chunks to retrieve, the usual approaches are:

Top-k: retrieve the k most similar chunks regardless of score (e.g., top 5)
Threshold: retrieve all chunks above a similarity score of 0.8
Combined: top-k, but only above a minimum threshold

How many you actually want depends on the task. If answers require cross-referencing multiple sources, 10–15 chunks may be necessary. If documents are self-contained and answers live in one place, 3–5 chunks are usually enough — more is noise.

A practical tip: query reformulation

In agentic architectures, the user’s raw question usually doesn’t go directly to the RAG tool. The agent reformulates it first to maximize retrieval quality.

Example:

User: What does procedure X specify for invoice registration?

Agent’s RAG query: invoice registration procedure requirements steps approved documents

The user’s question is conversational. The reformulated query is optimized for vector similarity — keywords, no filler. This is one of the concrete advantages of wrapping RAG in an agent rather than calling it directly.

One nuance worth knowing: because LLMs are probabilistic, the reformulated query can vary across calls even for the same user question. This means the retrieved chunks may differ too. This isn’t a problem — it’s a property. But it’s something to keep in mind when debugging retrieval behavior or comparing runs, especially as models get updated or deprecated.

What RAG doesn’t solve

RAG works well for factual retrieval from structured, relatively clean document sets. It struggles when:

Documents are poorly parsed (especially PDFs with complex layouts)
The relevant information requires synthesizing content across many chunks
Queries are ambiguous and the agent can’t reformulate them effectively

These failure modes aren’t reasons to avoid RAG — they’re the design constraints you need to know before you build.

What’s next

RAG extends the knowledge base at inference time. But there’s another way to give a model new knowledge: fine-tuning — baking it in permanently. The next post covers what fine-tuning actually is, when it makes sense, and how it compares to RAG. They solve different problems, and confusing the two leads to expensive mistakes.

If you have questions or want to suggest a topic, reach out.