Fine-tuning Basics

The idea

Pretraining teaches a model the broad patterns of language from enormous amounts of text. Fine-tuning teaches it something specific afterward: the tone of your support replies, the format of your legal summaries, the vocabulary of your medical specialty. Technically it is the same process as pretraining, gradient descent on a loss, continued on a smaller, curated dataset. Conceptually it is the difference between a general education and job training.

This works because of transfer learning. The pretrained model already understands grammar, facts, and reasoning patterns. Fine-tuning does not teach those from scratch; it steers existing capability toward the target behavior. That is why it needs far less data than pretraining: thousands of good examples can reshape behavior that took trillions of tokens to build.

What gets fine-tuned, and how

The straightforward approach updates all of the model's weights on the new data. This is full fine-tuning. It gives the most flexibility but needs the most memory: gradients and optimizer state for every parameter, which for large models means serious hardware. It also risks catastrophic forgetting, where the model loses general abilities while specializing. Train a model exclusively on legal documents for long enough and its casual conversation skills degrade.

Parameter-efficient methods avoid both problems by freezing most of the model and training only a small addition. The most popular is LoRA, low-rank adaptation, introduced in 2021. Instead of updating a weight matrix W directly, LoRA learns two small matrices A and B whose product approximates the desired change: the effective weights become W + BA, where the inner dimension is tiny compared to W. For a large model this cuts the trainable parameters to a fraction of a percent, which slashes memory needs and lets fine-tuning run on a single GPU. The adapter can also be swapped in and out, so one base model can serve many specialized variants.

In practice, full fine-tuning suits smaller models and cases where behavior must change deeply. LoRA and its relatives suit large models and most applied work. The quality gap between them is small for typical tasks.

The data matters more than the method

Fine-tuning lives or dies on its dataset. A few thousand excellent examples beat a million mediocre ones. Each example should show the exact behavior wanted: the input as the model will see it, the output as it should respond. Inconsistent examples teach inconsistency; the model faithfully learns whatever mixture it is given, including the bad habits.

A few practical numbers from common practice: one to three passes over the data (epochs) is typical for fine-tuning, far fewer than pretraining. Learning rates run smaller than pretraining, often in the range of 1e-5 to 1e-4, because large steps would wreck the pretrained weights. A held-out validation set is essential, since overfitting on a small fine-tuning set happens fast: the model starts parroting the training examples instead of generalizing from them.

The alignment stages

Modern assistants go through specialized rounds of fine-tuning with distinct purposes.

Supervised fine-tuning, or instruction tuning, trains on prompt-response pairs that demonstrate helpful behavior: questions with good answers, tasks with correct completions. This teaches the model the assistant format.

Reinforcement learning from human feedback takes a different approach. Human annotators rank the model's responses, a reward model learns those preferences, and the language model is optimized against the reward model. This is how models learn the subtler behaviors: which answer is most helpful, when to refuse, how to hedge appropriately. InstructGPT, the 2022 work that led to ChatGPT, established this pipeline.

Direct preference optimization is a newer alternative that skips the separate reward model and trains directly on the ranked pairs. It is simpler to run and has become common in open-model fine-tuning.

When to fine-tune, and when not to

Fine-tuning is the right tool when the desired behavior is hard to describe but easy to demonstrate: a house writing style, a specific output format, judgments that need many examples to convey. It bakes the behavior into the weights, so it works on every future input without lengthy prompts.

It is the wrong tool for knowledge. A fine-tuned model still has frozen weights after training; it cannot learn facts that change. Teaching a model your company's current product catalog through fine-tuning means retraining every time the catalog changes, and the model may still garble details. Retrieval-augmented generation handles changing knowledge better by fetching it at query time.

Prompting sits between the two. If a few examples in the prompt get the behavior you want, fine-tuning is unnecessary overhead. The usual progression is: try prompting first, add retrieval if knowledge is the gap, and fine-tune when the behavior itself needs to change.

The honest limits

Fine-tuning cannot fix a base model that lacks the underlying capability. If the pretrained model cannot reason about a domain, fine-tuning on that domain's examples teaches it to imitate the surface form, not the substance. It also inherits the base model's flaws, including biases and hallucination tendencies, which fine-tuning can soften but not remove. And every fine-tune is a snapshot: the moment the world changes, the specialized behavior starts going stale, and the pipeline runs again.