Overfitting and Regularization

Memorizing versus learning

A model that gets every training example right is not necessarily a good model. It might have learned the underlying pattern, or it might have simply memorized the answers. The difference shows up the moment it sees new data. The model that learned the pattern keeps working. The model that memorized falls apart.

This is overfitting. It happens when a model is flexible enough to fit not just the signal in the training data but the noise too: the typos, the coincidences, the quirks of that particular sample. A sufficiently large neural network can memorize its entire training set, labels and all, and still be useless for anything else. The training error goes to zero while the real-world error stays high, which is the worst kind of misleading metric.

The standard defense is to split data three ways. The training set is what the model learns from. The validation set is used to check progress and tune settings; the model never trains on it. The test set is locked away until the very end for one final honest measurement. If training accuracy climbs while validation accuracy stalls or drops, the model is overfitting, and you know it before deployment rather than after.

The bias-variance tradeoff

Underneath overfitting sits a genuine tension. A model that is too simple cannot capture the real pattern: it underfits, doing poorly on training and new data alike. A straight line cannot fit data that curves. A model that is too flexible captures the pattern plus the noise: it overfits, doing well on training data and poorly on new data.

Somewhere between them is the sweet spot, and its location depends on how much data you have. More data lets you afford more flexibility, because the noise averages out across many examples while the signal accumulates. This is one reason deep learning took off when large datasets appeared: the same network that overfits hopelessly on ten thousand images learns genuinely on ten million.

Regularization: constraining the fit

Regularization is the umbrella term for techniques that discourage overfitting, usually by limiting what the model is allowed to do. The common ones each attack the problem differently.

L2 regularization, also called weight decay, adds a penalty to the loss for large weights: the training objective becomes the original loss plus a constant times the sum of squared weights. This pushes the network toward small, diffuse weights instead of a few large ones that latch onto specific training examples. Large weights make the model's output swing wildly with small input changes, which is exactly the behavior that memorizes noise. Penalizing them smooths the learned function.

L1 regularization penalizes the sum of absolute weights instead. It has a side effect L2 lacks: it drives some weights all the way to zero, effectively removing connections. This produces sparser models and can serve as automatic feature selection.

Dropout takes a stranger approach. During training, each neuron is randomly switched off with some probability on each pass, so the network can never rely on any single neuron being present. The remaining neurons have to pick up the slack, which forces redundant, distributed representations. At test time all neurons are active, with their outputs scaled to match. Introduced by Srivastava and colleagues in 2014, dropout was one of the techniques that made the 2012-era deep networks trainable on real data.

Early stopping is the simplest of all. Watch the validation error during training and stop when it starts rising, even if the training error is still falling. The model's state at that point is kept. It costs nothing to implement and works surprisingly well.

Data augmentation fights overfitting by manufacturing more data. An image classifier can train on flipped, cropped, and color-shifted copies of each image. The model never sees the same exact input twice, which makes memorization pointless. For text, augmentation is harder, but paraphrasing and back-translation serve a similar role.

What does not count

A few things get confused with regularization. Getting more real data is the best fix for overfitting, but it is not regularization; it changes the problem rather than constraining the solution. Making the model smaller reduces overfitting, but it also reduces what the model can learn, which is a tradeoff rather than a technique. And cross-validation, training on different data splits to get a reliable performance estimate, measures overfitting without preventing it.

The modern wrinkle

Classical theory predicted a U-shaped curve: as models get bigger, test error falls, then rises as overfitting sets in. In practice, very large neural networks often show a second descent: past a certain size, test error starts falling again. These enormous models still fit their training data perfectly, but with so many parameters they find smooth, simple solutions among the many perfect fits rather than jagged memorized ones. The phenomenon, called double descent, does not mean overfitting is solved. It means the relationship between size and generalization is stranger than the textbooks suggested, and regularization still matters for everything that is not trained at massive scale.