Curriculum

Deep Learning · Sequential Models

Recurrent Neural Networks (RNN)

RNNs maintain a hidden state that persists across time steps, enabling sequential modelling. LSTMs and GRUs add gating mechanisms to capture long-range dependencies without vanishing gradients.

The idea

Standard RNNs update their hidden state at each time step: hₜ = tanh(Wₕ·hₜ₋₁ + Wₓ·xₜ + b). The hidden state carries a summary of all past inputs. But gradients through many time steps suffer the same vanishing/exploding problem as deep feedforward networks.

LSTM (Long Short-Term Memory) adds a cell state c and three gates: input (how much new information enters), forget (how much old state to clear), and output (how much cell state becomes the hidden state). Gates are sigmoid-gated, values near 1 let information through, near 0 they block it.

GRU (Gated Recurrent Unit) simplifies LSTM to two gates (reset and update), merging cell and hidden state. Slightly fewer parameters, often comparable performance. Transformers have largely superseded RNNs for text, but RNNs remain competitive in streaming inference, time-series, and resource-constrained settings.

hₜ = tanh(Wₕ·hₜ₋₁ + Wₓ·xₜ + b)

Vanilla RNN update. hₜ is the hidden state at step t; xₜ is the input.

What to hold onto

  • Backpropagation Through TimeBPTT unrolls the RNN across time steps and backprops as if it were a deep feedforward network. Gradient clipping is essential.
  • Bidirectional RNNRun two RNNs, one forward and one backward in time, concatenate hidden states. Used in sequence labelling (NER, speech recognition) where full context is available.
  • Sequence-to-SequenceEncoder RNN reads the input sequence into a context vector; decoder RNN generates the output sequence conditioned on it. Foundation of early NMT.

Live: Recurrent Neural Network State Memory

Type characters into the sequence. Watch the hidden state vector $h_t = \tanh(W x_t + U h_{t-1})$ evolve across time steps.

Hidden States h₀..hₜ

Quick answers

What is a Recurrent Neural Network?

An RNN is a neural network with a hidden state that persists across time steps, making it suitable for sequential data like text, audio, and time series. It processes inputs one step at a time, updating its state based on the current input and the previous state.

What is the advantage of LSTM over a vanilla RNN?

LSTMs use gating mechanisms (input, forget, output gates) to control information flow through a separate cell state. This allows gradients to flow over hundreds of time steps without vanishing, enabling the model to learn long-range dependencies that vanilla RNNs fail to capture.