LoRA Fine-tuning Explained

The problem LoRA solves

Full fine-tuning updates every parameter in a model. For a 175-billion-parameter model that means storing gradients and optimizer states for 175 billion values, which puts training out of reach for most teams. The LoRA paper (Hu et al., 2021) demonstrated that this full update is overkill: you can freeze the pretrained weights and train only a small added structure, getting performance on par with full fine-tuning while training far fewer parameters.

The empirical basis is a finding by Aghajanyan et al. (2020) that fine-tuning updates live in a low-dimensional subspace. In plain terms: although a model has billions of parameters, the useful change for a given task can be expressed with much less information. LoRA is a practical way to exploit that.

The math, plainly

Take a frozen weight matrix W of shape d by k. A normal fine-tuning run would learn a full update ΔW of the same shape. LoRA instead represents the update as a product of two thin matrices:

ΔW = (α / r) · B · A

Here A is r by k, B is d by r, and r (the rank) is a small number like 8, 16, or 64, far smaller than d or k. The number of trainable values drops from d × k to r × (d + k). For a 4096 by 4096 attention projection with rank 16, that is roughly 131,000 trainable parameters instead of 16.7 million: about a 128x reduction per layer.

The α term is a scaling factor (lora_alpha in the peft library). The update is multiplied by α / r, which keeps the effective learning rate stable as you change the rank: raising r adds capacity without silently changing the update magnitude. Both A and B are trained while W stays frozen.

A critical detail is initialization. The B matrix starts at zero and A starts from a small random distribution (Kaiming uniform by default in peft). At step zero the product B·A is zero, so the model behaves exactly like the pretrained base model and the adapter grows its influence gradually during training.

At inference time the adapter can be merged into the base weights: W' = W + (α/r)·B·A. Because the product is the same shape as the original matrix, the merged model runs with zero extra latency. This is the advantage LoRA has over adapter layers, which add sequential computation at inference.

What the numbers say

The original paper's headline results, using GPT-3 175B as the example: LoRA reduced trainable parameters by 10,000 times compared to Adam-based full fine-tuning and cut GPU memory requirements by 3 times, while matching or beating full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3. No inference latency penalty, because adapters merge into the base weights.

In practice, a rank-16 LoRA on a 7B model trains well under 1% of the parameters (often around 0.1 to 0.2% when targeting attention projections). That is why LoRA is swappable: one base model can hold many task-specific adapters, each a few megabytes, loaded on demand.

A minimal working example

This trains a LoRA adapter on an instruction dataset with the Hugging Face peft and trl libraries. Install with pip install transformers peft trl datasets torch.

```python

from datasets import load_dataset

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments

from peft import LoraConfig, get_peft_model

from trl import SFTTrainer

model_id = "meta-llama/Llama-3.2-1B"

model = AutoModelForCausalLM.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)

tokenizer.pad_token = tokenizer.eos_token

config = LoraConfig(

r=16, # rank of the update matrices

lora_alpha=32, # scaling factor, often 2x the rank

target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],

lora_dropout=0.05,

task_type="CAUSAL_LM",

)

model = get_peft_model(model, config)

model.print_trainable_parameters()

# trainable params: ~2M || all params: ~1.2B || trainable%: ~0.2%

dataset = load_dataset("yahma/alpaca-cleaned", split="train[:1000]")

args = TrainingArguments(

output_dir="lora-out",

per_device_train_batch_size=4,

num_train_epochs=2,

learning_rate=2e-4,

fp16=True,

logging_steps=20,

)

trainer = SFTTrainer(model=model, args=args, train_dataset=dataset, max_seq_length=512)

trainer.train()

model.save_pretrained("lora-adapter")

```

save_pretrained stores only the adapter weights, not the base model. That checkpoint is typically a few megabytes. To load it later, build the same base model and attach the adapter with PeftModel.from_pretrained(base_model, "lora-adapter"), or bake it into the weights permanently with model.merge_and_unload().

QLoRA: when the base model will not fit in memory

LoRA reduces trainable parameters, but the frozen base model still sits in GPU memory at full precision. A 65B model needs roughly 130GB just to load in 16-bit, which is beyond a single consumer or workstation GPU. QLoRA (Dettmers et al., 2023) fixes this by quantizing the base model to 4-bit before training.

QLoRA introduces three pieces: NF4, a 4-bit data type designed for the normal distribution of neural network weights; double quantization, which quantizes the quantization constants themselves to save another ~0.37 bits per parameter; and paged optimizers, which spill optimizer memory spikes to CPU RAM instead of crashing. Gradients flow through the frozen 4-bit weights into LoRA adapters kept at 16-bit precision.

The paper's result: fine-tuning a 65B model on a single 48GB GPU with no quality loss versus 16-bit full fine-tuning. Their Guanaco models reached 99.3% of ChatGPT's performance on the Vicuna benchmark after 24 hours of fine-tuning on one GPU. QLoRA is now the default approach whenever GPU memory is the binding constraint. The setup in code is a quantized base model plus a standard LoRA config:

```python

import torch

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

from peft import LoraConfig, get_peft_model

bnb = BitsAndBytesConfig(

load_in_4bit=True,

bnb_4bit_quant_type="nf4",

bnb_4bit_compute_dtype=torch.bfloat16,

bnb_4bit_use_double_quant=True,

)

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-8B", quantization_config=bnb)

model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"))

```

Note target_modules="all-linear", which applies LoRA to every linear layer instead of just attention projections. The QLoRA results used all-layer targeting, and matching that configuration gets closer to full fine-tuning quality than attention-only adapters.

Choosing rank, alpha, and target modules

Start with rank 16 and lora_alpha 32. Raise the rank when the task is complex or the dataset is large: 32 or 64 for hard tasks, 8 for simple style shifts on small datasets. Keep alpha at roughly 2x the rank to preserve the scaling behavior described earlier. If you raise the rank and see no improvement on your eval set, the extra capacity is not helping; go back down.

Target modules matter more than rank. The LoRA paper targeted query and value projections in attention; the QLoRA paper targeted all linear layers and the peft docs note this can match fully fine-tuned quality. More targets means more trainable parameters, which costs memory and time, so treat attention-only as the cheap default and all-linear as the quality option.

Learning rate deserves one note: LoRA runs typically use higher learning rates than full fine-tuning, commonly around 1e-4 to 3e-4, because only a tiny parameter set is moving. As always, this is a starting point, and your eval set decides.

When LoRA underperforms full fine-tuning

LoRA is a constrained update: the change must fit inside a low-rank subspace. Tasks where it falls short are the ones that need large, high-dimensional changes to the model. The clearest case is teaching a model large amounts of genuinely new knowledge from scratch, such as pretraining-style continued training on a domain the base model barely saw. If the base model has no scaffolding for the new material, a rank-16 adapter cannot build it.

It also struggles when the target distribution is very different from the pretrained one and the training data is large enough that full fine-tuning is affordable. In that regime the low-rank constraint stops being a feature and starts being a bottleneck. A practical rule: when QLoRA-style all-linear adapters at rank 64 still underperform, and you can afford the compute, try full fine-tuning on the same data and compare on evals. Do not assume LoRA is always enough; measure.

Finally, watch adapter stacking. Multiple LoRA adapters merged or composed on one base model can interact in ways that degrade quality, and the order of merging matters. Test composed adapters on evals rather than assuming they combine cleanly.