Loss Functions

One number to minimize

Training needs a way to say how wrong the model currently is, as a single number that gets smaller as predictions improve. That number is the loss. The entire training loop, forward pass, backpropagation, gradient descent, exists to drive it down. Choose the loss well and the model learns what you want. Choose it badly and the model learns something else with great dedication.

The loss must be differentiable, meaning you can compute how a small change in each weight would change the loss. This rules out the most intuitive measures. Classification accuracy, the fraction of examples gotten right, is what you actually care about, but it changes in jumps: nudging a weight slightly almost never flips a prediction, so the gradient is zero almost everywhere and training goes nowhere. The loss is a smooth stand-in that correlates with what you want while giving useful gradients everywhere.

Mean squared error

For regression, predicting a number, the standard loss is mean squared error: the average of the squared differences between predictions and targets.

```

MSE = (1/n) * sum((y_i - predicted_i)^2)

```

Squaring does two things. It makes all errors positive so they cannot cancel out, and it punishes large errors disproportionately: an error of 4 contributes 16 times as much as an error of 1. This matches many real goals, where being slightly off is fine but being wildly off is bad.

MSE has a clean gradient and a clear geometric meaning, the average squared distance. Its weakness is sensitivity to outliers. One bad label with a huge error can dominate the loss and drag the whole model toward it. When outliers are expected, mean absolute error is sometimes used instead, though its gradient has a kink at zero that makes optimization fussier.

Cross-entropy

For classification, the standard is cross-entropy. The model outputs a probability for each class, usually via softmax, and the loss measures how much probability it assigned to the correct class:

```

Cross-entropy = -sum(y_i * log(predicted_i))

```

Since the true label y is 1 for the correct class and 0 elsewhere, this simplifies to the negative log of the predicted probability for the right answer. If the model assigns 90% to the correct class, the loss is small. If it assigns 1%, the loss is large. The logarithm means the penalty grows steeply as confidence in the wrong answer rises: being confidently wrong is punished far more than being uncertain.

For yes-or-no classification with a single probability output, the binary version is:

```

Loss = -(y log(p) + (1 - y) log(1 - p))

```

Cross-entropy comes from information theory, where it measures the extra bits needed when you encode events with the wrong probability distribution. You do not need the theory to use it, but it explains why the loss behaves well: it is zero only when the model's probabilities exactly match reality, and it grows smoothly as they diverge.

Loss for language models

A language model is a classifier over its whole vocabulary at every position: given the text so far, it assigns a probability to each possible next token. The loss is cross-entropy between that distribution and the actual next token, averaged over all positions. This single objective, predict the next token, is what trains models like GPT.

Perplexity is the usual way to report this loss in human-friendly terms. It is the exponential of the average cross-entropy, and you can read it loosely as the model's average uncertainty: a perplexity of 20 means the model is as confused as if it were choosing uniformly among 20 options at each step. Lower is better. Perplexity is useful for comparing models on the same data and tokenizer, and meaningless across different ones.

Matching the loss to the task

The pattern is consistent: regression gets squared error, classification gets cross-entropy, ranking gets pairwise losses, and so on. The choice encodes what "wrong" means for the task. Using MSE for classification works after a fashion but gives weak gradients when the model is confidently wrong, which is exactly when you want the strongest signal. Using cross-entropy for regression makes no sense at all, since there are no probabilities involved.

In practice, models often train on composite losses: several terms added together, each capturing something the others miss. A translation model might combine cross-entropy on the target words with a penalty for repeating phrases. A generative image model might combine pixel-level error with a perceptual term that measures similarity the way human vision does. Each term needs a weight, and tuning those weights is part art, part experiment. When one term dominates, the model optimizes that term and ignores the rest, so balancing them matters as much as choosing them.

There is a subtle trap here. The loss defines what the model optimizes, and models optimize exactly what you measure, including the parts you did not intend. A recommendation system trained to maximize predicted clicks will learn to maximize clicks, even if that means promoting outrage. The loss is a specification of the goal, and like all specifications, it captures what you wrote down rather than what you wanted. Getting the loss right is part of getting the objective right.