What is RAG

The problem it solves

Language models have two hard limits. Their knowledge is frozen at training time, so they know nothing about events since, and they know nothing about your private data: your company's documents, your codebase, your product catalog. Ask about any of it and the model either confesses ignorance or, worse, invents an answer.

Retraining the model on new information is slow and expensive, and it does not work at all for data that changes daily. Retrieval-augmented generation takes a different approach: when a question arrives, fetch the relevant documents first, put them in the prompt, and let the model generate its answer from that provided context. The model's weights never change. The knowledge comes along for the ride at query time.

The name comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research and University College London, presented at NeurIPS. They paired a pretrained sequence-to-sequence model with a dense index of about 21 million Wikipedia passages, retrieved with a learned neural retriever, and showed the combination beat purely parametric models on knowledge-intensive question answering. The paper framed it as combining parametric memory, the model's weights, with non-parametric memory, an external index that can be updated without retraining.

The pipeline

A RAG system runs the same sequence for every query.

First, documents are prepared offline. Long documents get split into chunks, typically a few hundred tokens each, because retrieval works best on focused passages and because chunks must fit in the context window. Each chunk is converted to a vector by an embedding model and stored in a vector database alongside the original text.

When a question arrives, it is embedded with the same model, and the index returns the chunks whose vectors are closest to the question's vector, usually by cosine similarity. Those chunks, the original text, are inserted into the prompt ahead of the question. The language model then generates its answer conditioned on both.

The retrieval half is what makes it RAG. Any language model can do the generation step. Finding the right passages, quickly, from millions of candidates, is the actual engineering.

Why chunking and embeddings matter

Chunk size is a real tradeoff. Chunks that are too large dilute the relevant content with irrelevant text and waste the context window. Chunks that are too small lose the surrounding context that makes a passage meaningful; a paragraph that says "he signed it" is useless without knowing who "he" is. Overlapping chunks, where each chunk shares some text with its neighbors, are a common compromise.

The embedding model determines what "relevant" means. It must be good at the domain's language: a general-purpose embedder may miss the significance of specialized terminology. Since both documents and queries pass through the same model, changing the embedder means re-embedding the entire collection, so the choice is worth getting right early.

Beyond the basics

The naive pipeline, embed everything and take the top matches, works well enough to be useful and badly enough to invite improvement. Common upgrades include hybrid search, which combines vector similarity with traditional keyword matching so that exact terms like product codes are not missed. Reranking passes the initial candidates through a more careful model that scores each one against the query directly, trading speed for accuracy on a small set. Query rewriting has the model reformulate the user's question into better search queries before retrieval. And metadata filtering restricts the search to the right subset, documents from the right year, or the right department, before similarity is even computed.

On the generation side, the standard advice is to instruct the model to use only the provided context and to cite which passage each claim came from. Citations do not guarantee correctness, but they make answers checkable, which changes the trust calculation entirely.

RAG versus fine-tuning

The two techniques solve different problems and are often confused. Fine-tuning changes behavior: the model's style, format, and judgment on recurring tasks. RAG supplies knowledge: facts the model never saw in training or that change too often to bake into weights.

A support bot needs both, arguably. Fine-tuning teaches it the company's voice and how to structure answers. RAG gives it the current documentation to answer from. But if you must pick one, the deciding question is whether the gap is knowledge or behavior. If the model would answer correctly given the right documents, that is a retrieval problem. If it has the facts and still responds wrongly, that is a training problem.

RAG also wins on updates. Editing the knowledge means editing the documents and re-embedding them, which takes minutes. Updating a fine-tuned model's knowledge means another training run. For anything that changes regularly, retrieval is the sane default.

The limits

RAG reduces hallucinations; it does not eliminate them. The model can still misread a passage, combine two passages into a false claim, or answer from its parametric memory when the retrieved context is thin. Retrieval quality bounds everything: if the right document is not in the top results, the model answers without it, and no prompt instruction fully prevents that.

There is also a garbage-in problem. RAG over a messy, outdated, or contradictory document collection produces answers grounded in mess. The system is only as trustworthy as its corpus, which makes document curation part of the engineering, not someone else's problem.

Finally, RAG does not improve reasoning. A model that cannot follow a multi-step argument will not learn to by reading more documents. Retrieval supplies the raw material. What the model does with it is still bounded by what the model can do.