Curriculum

Deep Learning · Training Objectives

Loss Functions & Optimisation

A loss function quantifies how wrong the model is. The training loop minimises it. MSE dominates regression; Cross-Entropy dominates classification. Choice of loss shapes the learned representation.

The idea

The loss function is the mathematical interface between model predictions and ground truth. Every design decision in training—learning rate, batch size, regularisation—is calibrated around making the loss small on held-out data.

Mean Squared Error (MSE): L = (1/n)Σ(ŷᵢ−yᵢ)². Differentiable everywhere, penalises large errors quadratically. Sensitive to outliers. The square makes the loss landscape bowl-shaped for linear models.

Binary Cross-Entropy: L = −[y·log(p) + (1−y)·log(1−p)]. Derived from maximum likelihood for Bernoulli outputs. Pairs naturally with a sigmoid output neuron.

Categorical Cross-Entropy: L = −Σ yᵢ·log(pᵢ). Pairs with Softmax. When the target is one-hot and the output is Softmax, this is negative log-likelihood of the correct class.

Huber Loss: quadratic for |error|≤δ, linear beyond. More robust to outliers than MSE while remaining smooth at zero—a practical compromise for regression in noisy settings.

L = −(1/n) Σᵢ [yᵢ log(ŷᵢ) + (1−yᵢ) log(1−ŷᵢ)]

Binary Cross-Entropy loss—minimised when ŷ matches y. Equivalent to maximising log-likelihood.

What to hold onto

  • KL DivergenceKL(P||Q) = Σ P(x)·log(P(x)/Q(x)). Measures how much distribution Q diverges from P. Cross-entropy = KL divergence + entropy of P.
  • Focal LossReweights cross-entropy so the model focuses on hard, misclassified examples. Crucial for one-stage object detectors like RetinaNet.
  • RegularisationL2 (weight decay): adds λ·||w||² to loss. L1: adds λ·||w||₁, promoting sparsity. Dropout is implicit regularisation.

Live: Loss Function & Outlier Penalty Curve

Compare MSE ($e^2$) vs Huber Loss vs L1 MAE ($|e|$) as prediction error varies. Notice how Huber transitions from quadratic to linear to resist outliers.

Loss: 1.440

Quick answers

What is a loss function in deep learning?

A loss function measures the discrepancy between a model's prediction and the true target. Training minimises it with gradient descent. The choice of loss determines what "correct" means: MSE penalises squared errors for regression, while Cross-Entropy measures probability divergence for classification.

When should you use Huber Loss instead of MSE?

Huber Loss is preferred when the training data contains outliers. It behaves like MSE for small errors (smooth and differentiable at zero) but like MAE for large errors (linear, less sensitive to outliers). This makes it more robust than MSE without sacrificing gradient-friendliness near zero.