how RAG works
for document retrieval

Back in 2023, I built a first version of this at home in Edmonton for a specific problem. Alberta Wildfire was publishing a new PDF bulletin every day during a bad spring, and reading a fresh twenty‑page report every morning to answer “is anything burning near us?” was tedious. I stood up Wildfire Insight Engine — Streamlit for the UI, LangChain to orchestrate, FAISS as the vector store, an OpenAI model behind it all — and pointed it at every day’s bulletin PDF. It ran in an afternoon and made a beautiful demo: pick a day, type a plain question, get a highlighted paragraph from that day’s report. It also happily invented answers when the bulletin didn’t cover the question, because nothing in that pipeline told it to refuse. That gap is what separated a party trick from a system anyone else could actually trust.

Two years on, the architecture has hardened. Streamlit is still fine for prototyping, but the vector stores that stuck around are the boring, indexable ones. LangChain and its cousins got quieter about what they promise. And the discipline that actually matters — a relevance floor, honest refusal, cited answers — became the price of admission rather than a stretch goal. This piece is what remained after the noise settled.

Every organization that has ever asked "what if we let people ask our documents questions?" ends up building the same thing. Not because it’s the only option, but because five or six years of open‑source work have converged on one shape. It has a name: Retrieval‑Augmented Generation, or RAG, and the reference implementation lives in a hundred GitHub repos wearing slightly different clothes.

This piece is the high‑level map. Five moving parts, one honest rule, three ways to deploy. If you have read a LangChain, LlamaIndex, or Haystack README, none of this will surprise you. If you haven’t, this is the shortest path to understanding what those frameworks are doing behind the curtain.

01the pipeline, in five boxes

Every RAG system is these five stages. Names differ across frameworks; the shape does not.

Stages 01–03 run once when documents change — call this ingestion. Stages 04–05 run for every user question — call this inference. Keeping the two loops separate is the single biggest architectural decision.

01 · chunk

Documents don’t fit in a language model’s context window, and even if they did, the model reads more clearly when it sees only what matters. You break each document into passages of roughly 300–800 tokens, usually with some overlap so ideas that straddle a chunk boundary are still findable. Every chunk keeps a link back to its source: file name, page number, section, version.

02 · embed

An embedding model reads each chunk and outputs a vector — a long list of numbers — that captures its meaning. Chunks about the same topic land near each other in vector space, regardless of the exact words they use. Common choices: OpenAI’s text‑embedding‑3, Cohere’s embed models, or open‑source options like all‑MiniLM or bge running locally.

03 · store

Vectors go into a database built for similarity search. The database keeps the vector, the chunk text, and the source metadata together, so a retrieved chunk can always cite its origin. Common choices: pgvector if you already run Postgres, Chroma or Qdrant for standalone open source, Pinecone or Weaviate if you want managed cloud.

04 · retrieve

When a user asks a question, you embed the question with the same model, then ask the database for the top‑K most similar chunks — usually 3 to 10. Modern setups add a second pass: a reranker reads the top‑K and reorders them by real relevance, discarding matches that were vector‑similar but semantically off.

05 · generate

The retrieved chunks and the original question go into a prompt that tells the model: answer the question using only what is between these delimiters, cite your sources, and say “I don’t know” if the answer isn’t there. The model writes a short, sourced answer. That is the whole trick.

02a worked example · the fable

Abstract pipelines are hard to trust. Here are the same five stages processing one very small document, end to end.

chunk · cut into passages

embed · turn each chunk into a vector

An embedding model reads each chunk and outputs a vector. Real embeddings are 768 to 3072 dimensions; three are shown below for illustration.

c1  →  [ 0.42, -0.18,  0.66, ...]
c2  →  [ 0.11,  0.79, -0.24, ...]
c3  →  [-0.34,  0.62, -0.08, ...]   // "the danger" - matters later
c4  →  [ 0.71, -0.45,  0.02, ...]
c5  →  [ 0.28,  0.14,  0.83, ...]

store · put the vectors in a database

Every row keeps three things together: the vector, the original text, and the metadata that will let the answer be cited.

retrieve · find the closest chunks at query time

generate · answer only from what was retrieved

the refusal · what “I don’t know” looks like

Now ask a question the document doesn’t answer.

Five sentences of source, five pipeline stages, one grounded answer, and one honest refusal. Every real RAG deployment is this same shape at larger scale.

03where to run it · three shapes

The pipeline above is engine‑agnostic. What changes across deployments is where each box runs. There are three sensible shapes.

04the one rule that keeps it honest

A RAG pipeline that always answers is a RAG pipeline that lies. The system must be engineered to refuse when retrieval turns up nothing useful. That refusal has three moving parts:

The trap for most first builds is skipping the threshold in step one. Without it, a nonsense question retrieves nonsense chunks, and the LLM — being fluent — writes a nonsense answer that sounds authoritative. Every serious RAG system has a floor below which it says nothing.

05the common stack, named

You do not have to write any of this from scratch. Almost every serious RAG deployment picks from the same short list:

06why this shape wins

The reason every RAG project on GitHub ends up looking like the diagram in section 01 is not fashion. It is that RAG separates two things that have to be separated: knowledge (which changes constantly and must be citable) and language ability (which is stable and can be rented). A fine‑tuned model fuses them, and every change to your documents means retraining. RAG keeps them apart, and every change to your documents is one reindex away.

That is also why the boring parts — the chunking rule, the similarity threshold, the citation format — matter more than the choice of LLM. A great model with sloppy retrieval writes fluent lies. A modest model with rigorous retrieval writes short, correct, cited answers. The second is what you want.

07closing

RAG is the reference architecture for grounding a language model in your data. It is boring, well‑understood, and almost commoditized. That is its strength: you can pick any framework, any vector DB, any LLM, and the pipeline still looks the same. What separates the demos from the systems is not the model you chose. It is the discipline of the retrieval floor, the honesty of the prompt, and the log that lets you prove where every answer came from.

the documents own the truth. the AI only helps them speak.