Curriculum

Topic 03 · Neural Networks

Deep Learning

Stack simple functions deep enough, put a non-linearity between each one, and the composition can approximate almost anything. Backpropagation finds the weights without hand-solving.

The idea

A single layer computes a weighted sum and bends it. Stack those and each layer gets to work on the representation the previous one produced rather than on raw input. Early layers in a vision network find edges; later ones find faces. Nobody specified either.

Backpropagation is the chain rule applied efficiently. Run the input forward, measure the error, then walk backwards multiplying local derivatives to learn how much each weight contributed to the mistake.

Without a non-linearity between layers, the whole stack collapses: a composition of linear maps is just another linear map. ReLU is the usual choice—cheap and does not saturate on the positive side.

∂L/∂w₁ = (∂L/∂a₃)(∂a₃/∂a₂)(∂a₂/∂w₁)

The chain rule applied layer-by-layer from the loss backwards.

What to hold onto

  • Activation functionsReLU is default. Sigmoid and tanh squash into bounded ranges and saturate, causing vanishing gradients in deep stacks.
  • Vanishing gradientsMultiply many small derivatives and the signal reaching early layers approaches zero. Residual connections give the gradient a shortcut.
  • RegularisationDropout, weight decay and early stopping all exist because a network with millions of parameters can memorise the training set outright.

Live: Neural Network Signal Propagation

Watch activation signals propagate across 4 layers. Sliders adjust input values $x_1, x_2$; connection thickness indicates weight magnitude.

Output ŷ: 0.812

Quick answers

What is Deep Learning?

Deep Learning uses neural networks with many layers to learn hierarchical representations from data. Each layer transforms its input into progressively abstract features, enabling the system to recognise patterns in images, text, and audio without hand-crafted feature engineering.

Why does Deep Learning need non-linear activations?

Without non-linear activation functions, stacking layers achieves nothing—any composition of linear functions is itself linear. Non-linearities like ReLU allow deep networks to approximate arbitrary complex functions, which is the source of their expressive power.