Curriculum

Deep Learning · Architecture Design

Deep Network Architectures

Deep architectures stack dozens to hundreds of layers. Residual connections, batch normalisation, and skip pathways solve the training instability that plagued early deep stacks.

The idea

Going deeper improves representational capacity—but only if gradients can flow. In a 100-layer stack, the gradient signal is a product of 100 Jacobian matrices. If each has a spectral radius below 1, the signal vanishes exponentially; above 1, it explodes.

Residual Networks (ResNets) add a direct identity shortcut around every two-layer block: y = F(x) + x. Gradients flow through the shortcut unchanged, training 152-layer networks that win ImageNet. The key insight is that learning a residual is easier than learning a direct mapping.

Batch Normalisation normalises layer activations to zero mean and unit variance within each mini-batch, then applies learnable scale and shift. This stabilises the distribution of inputs to each layer, allowing higher learning rates and reducing sensitivity to initialisation.

y = F(x, {Wᵢ}) + x

ResNet residual block: the shortcut ensures identity is the minimum achievable function.

What to hold onto

  • Skip ConnectionsDenseNet connects every layer to every subsequent layer. Wide ResNets increase width instead of depth. Both improve gradient flow.
  • Layer NormalisationLN normalises across feature dimensions, not batch. Preferred in Transformers where batch sizes can be very small.
  • Depth vs. WidthDeeper networks learn more abstract features; wider networks have greater raw capacity. EfficientNet searches for the optimal balance.

Live: Residual Block & Skip Connection Flow

Compare gradient signal propagation in a Plain 10-layer network vs a ResNet with residual skip connections $mathcal{F}(x) + x$.

Input Gradient: 0.892 (Preserved)

Quick answers

What are residual connections in deep networks?

Residual connections are shortcut paths that add a layer's input directly to its output, allowing gradients to bypass layers during backpropagation. Introduced in ResNet, they enable training of very deep networks (100+ layers) by preventing vanishing gradients.

What is Batch Normalisation?

Batch Normalisation normalises a layer's activations to zero mean and unit variance across each mini-batch, then re-scales with learnable parameters. This stabilises training, allows higher learning rates, and acts as a mild regulariser.