Curriculum

Topic 05 · Attention Mechanism

Transformers & Self-Attention

Transformers replaced RNNs for sequence modelling. Self-attention relates every token to every other in parallel, enabling global context with O(n²) compute—the foundation of GPT, BERT, and modern LLMs.

The idea

The transformer's core operation is scaled dot-product attention: each token computes a query Q, and every other token produces keys K and values V. Attention weights are softmax(QKᵀ/√d), and output is the weighted sum of V.

Multi-Head Attention runs h attention heads in parallel, each learning different relational patterns. The heads' outputs are concatenated and linearly projected. Position is injected via sinusoidal positional encodings or learned embeddings, since attention itself is permutation-invariant.

Attention(Q,K,V) = softmax(QKᵀ / √dₖ) · V

Scaled dot-product attention. √dₖ prevents dot products from growing so large that softmax saturates.

What to hold onto

  • Encoder vs. DecoderBERT is encoder-only (bidirectional, masked language model). GPT is decoder-only (causal, autoregressive). T5 uses both.
  • KV CacheAt inference, keys and values for past tokens are cached to avoid recomputation. Halves the memory needed for long-context generation.
  • Flash AttentionTiling-based exact attention that avoids materialising the full N×N matrix, reducing memory from O(n²) to O(n). Essential for long contexts.

Live: Multi-Head Self-Attention Matrix Viewer

Hover tokens to view query-key attention distribution $A = \text{softmax}(QK^T / \sqrt{d_k})$.

Self-Attention Heatmap

Quick answers

What is a Transformer?

A Transformer is a neural network architecture based on self-attention mechanisms that model relationships between all input tokens simultaneously. Unlike RNNs, Transformers process sequences in parallel and capture global context, making them the dominant architecture for NLP, vision, and multimodal tasks.

What is self-attention?

Self-attention allows each token in a sequence to compute a weighted mixture of all other tokens' representations. The weights are determined by learned query-key dot products, enabling the model to focus on relevant context regardless of distance. This replaces the sequential bottleneck of RNNs.