Curriculum

Deep Learning · Gradient Computation

Backpropagation

Backpropagation is the chain rule of calculus applied efficiently through a computation graph. It computes the gradient of the loss with respect to every weight in O(weights) time.

The idea

Backprop works backwards from the loss. At each layer it computes the local gradient (∂loss/∂output of this layer) and multiplies by the upstream gradient passed from the layer above (chain rule). Two passes per training step: one forward (compute activations and loss), one backward (compute and accumulate gradients).

The key equation at layer l: δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾)ᵀ δ⁽ˡ⁺¹⁾ ⊙ σ'(z⁽ˡ⁾), where δ is the "error signal" and σ' is the derivative of the activation. Then: ∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾(a⁽ˡ⁻¹⁾)ᵀ.

Modern frameworks (PyTorch, JAX, TensorFlow) implement automatic differentiation: they record the forward computation as a dynamic graph and replay it in reverse to accumulate gradients. The programmer never writes backprop by hand.

δ⁽ˡ⁾ = ((W⁽ˡ⁺¹⁾)ᵀ δ⁽ˡ⁺¹⁾) ⊙ σ'(z⁽ˡ⁾)

Error signal at layer l. ⊙ is element-wise multiplication. Multiplied by upstream delta and local activation derivative.

What to hold onto

  • Computational ComplexityBackward pass takes roughly the same compute as the forward pass. Total: O(2×forward) per training step.
  • Gradient AccumulationTo simulate large batch sizes on limited memory, accumulate gradients over N mini-batches before applying the update.
  • Gradient ClippingIf gradient norms exceed a threshold, scale them down. Prevents exploding gradients in RNNs and LLM fine-tuning.

Live: Computational Graph Backprop Tracer

Trace the backward pass adjoint calculation $\frac{\partial L}{\partial x_i} = \sum \frac{\partial L}{\partial z_j} \frac{\partial z_j}{\partial x_i}$ via the chain rule.

Forward values Backward Gradients (Adjoints)

Quick answers

What is Backpropagation?

Backpropagation is an efficient algorithm for computing the gradient of a loss function with respect to all network parameters. It applies the chain rule backwards through the computation graph, enabling gradient-based optimisation of neural networks with millions of parameters.

What is the vanishing gradient problem?

During backprop, gradients are multiplied by derivatives at each layer. If activations like sigmoid produce derivatives near zero (saturation), multiplying many small values causes the gradient to shrink exponentially, making early layers learn negligibly slowly or not at all.