Fine-tuning vs RAG

Two different ways to teach a model your data

Retrieval-augmented generation (RAG) keeps the model weights frozen and hands the model relevant documents at inference time. The model looks things up instead of memorizing them. Fine-tuning changes the weights themselves, baking new behavior into the parameters. Both start from the same place: a base model that knows general things but not your specifics.

The difference is architectural. RAG adds a retrieval step before generation: embed the query, search a vector store, insert the top chunks into the prompt, generate the answer. Fine-tuning runs gradient updates on example input/output pairs so the model produces the right output without being shown the source material. Nothing about the two is mutually exclusive, which is where most production systems end up.

What each one changes

RAG changes what the model sees. It is a data problem: chunk your documents, build an index, write retrieval code, handle the cases where retrieval returns the wrong chunk. The model itself stays untouched, so base capabilities are preserved and the knowledge base can be updated by re-indexing instead of retraining.

Fine-tuning changes how the model behaves. It is a training problem: collect example pairs, run a training job, evaluate the result. Once trained, the knowledge or behavior is inside the weights. Updating it means retraining, not re-indexing. OpenAI's own optimization guide puts evals and prompt engineering ahead of fine-tuning in the loop for a reason: measure first, prompt with context, and only reach for weight updates when the simpler steps fall short.

When retrieval is the right call

Reach for RAG when the answers must come from documents that change. Prices, policies, product catalogs, internal wikis, support articles: any of these can be re-indexed in minutes while retraining a model for each change is impractical. RAG also wins when you need provenance. Every answer can cite the chunk it came from, which matters for compliance reviews and debugging wrong answers.

RAG is also the lower-commitment starting point. No training data to curate, no GPU time, and you can evaluate it by reading retrieved chunks before trusting the generator. When people say RAG is cheaper, this is mostly what they mean: the cost is engineering and embedding, not training infrastructure.

When you need to update the weights

Fine-tuning earns its cost when the problem is behavior, not information. A model that must always respond in a specific format, classify into your internal label set, write in a house style, or reliably emit structured JSON for hundreds of edge cases: these are behavioral patterns that are hard to express as retrieved context. OpenAI's docs note two practical advantages that fall out of this: you can use shorter prompts with fewer examples, which cuts token costs at scale and reduces latency, and you can distill a task into a smaller, cheaper model that performs like a larger one on that narrow task.

Fine-tuning is also worth it when you have more examples than fit in a prompt. If the pattern needs thousands of demonstrations to become reliable, retrieval cannot inject them all at generation time. Weight updates can absorb them.

There is a third case worth naming honestly: tasks where fine-tuning teaches genuinely new capabilities, like domain reasoning over medical guidelines or legal case law. This is where OpenAI's reinforcement fine-tuning (RFT) fits, and it is also where the data and compute demands are highest.

The decision framework

Ask these questions in order.

1. Does the task change how the model should act, or only what it should know? Behavior points to fine-tuning. Knowledge points to RAG.
2. Does the knowledge change often enough that retraining would be a burden? Weekly or daily changes point to RAG. Stable knowledge lowers the bar for fine-tuning.
3. Do you need citations back to source documents? That is RAG territory; weights do not cite sources.
4. How many examples would it take to demonstrate the behavior? Dozens of edge cases fit in prompts; thousands do not.
5. What is the latency budget? RAG adds retrieval time plus longer prompts. A fine-tuned model on short prompts is leaner at inference, and OpenAI explicitly flags token savings and lower latency as fine-tuning benefits.
6. Do you have labeled training data and an eval set? Fine-tuning without evals is guesswork. If you cannot measure the baseline, you cannot justify the training job.

For each of these, the answer "both" is valid and common. A fine-tuned model with retrieval is a standard production pattern: the weights carry style, format, and domain judgment, while retrieval supplies the facts that change.

Cost and latency tradeoffs

Up front, RAG is cheaper. Indexing embeddings and running a vector store costs far less than a training run, and open-source vector databases make the infrastructure affordable. At inference, RAG is usually more expensive per query: every request pays for embedding the query, searching the index, and feeding retrieved chunks into the context window. Token usage grows with the number and size of chunks.

Fine-tuning reverses this. Training costs come first, and they range from modest (a LoRA run on a few GPUs) to large (full fine-tuning runs). Per query, a fine-tuned model is typically cheaper and faster than a RAG system because the prompt is shorter: no retrieved chunks, no embedding call, no search round trip. The break-even point depends on query volume, prompt lengths, and how often you retrain. A high-volume, narrow task usually tips toward fine-tuning. A low-volume, broad knowledge base usually tips toward RAG.

The honest middle: hybrid systems

Most serious systems do both. Fine-tune for tone, format, and domain behavior; retrieve for facts. A customer support bot can be fine-tuned on the company's voice and escalation rules while retrieving the latest policy documents at runtime. The division maps cleanly to what each technique is good at: weights for patterns that repeat across queries, retrieval for content that varies or changes.

One practical caution for hybrids: evaluate the pieces separately. If an answer is wrong, you need to know whether retrieval returned the wrong chunk or the model misused the right one. Log retrieved chunks with every answer during development.

A note on the OpenAI fine-tuning platform

As of 2026, OpenAI is winding down its hosted fine-tuning platform: it is no longer open to new users, existing users retain access for a transition period, and fine-tuned models keep working until their base models are deprecated. If your plan depends on hosted fine-tuning from OpenAI, check the current timeline on their docs page and have a migration plan. Alternatives include fine-tuning open-weight models yourself (LoRA and QLoRA make this practical on modest hardware) or fine-tuning APIs from other providers.

What to do next

Start with evals that measure what you care about, then prompt engineering with retrieved context, then fine-tuning only where the simpler steps fail. This is the order OpenAI recommends, and it is the order that wastes the least time: each step tells you whether the next one is necessary.