Curriculum

Deep Learning · Optimisation

Gradient Descent & Stochastic Variants

Gradient descent iteratively moves parameters in the direction that reduces loss. SGD, momentum, Adam and its variants control speed, stability and generalisation of the training loop.

The idea

Gradient Descent (GD) moves parameters opposite the gradient: θ ← θ − η·∇L(θ). Full-batch GD uses the entire dataset per step—stable but too slow for large datasets.

Stochastic GD (SGD): sample one example per step. Noisy but fast and can escape sharp minima. Mini-batch SGD: sample B examples. Balances noise and compute efficiency. B∈{32,64,128,256} is standard.

Momentum: exponential moving average of past gradients. Accelerates in persistent directions, dampens oscillations: v ← βv + ∇L; θ ← θ − ηv.

Adam: tracks first moment (mean) m and second moment (variance) v of gradients, with bias correction. Effective learning rate adapts per parameter: θ ← θ − η·m̂/√v̂+ε. Works well out-of-the-box; AdamW fixes weight decay coupling.

θ ← θ − η · ∇θ L(θ)

Vanilla gradient descent parameter update. η is the learning rate.

What to hold onto

  • Learning Rate SchedulesCosine annealing, step decay, linear warmup + cosine. LR is the most sensitive hyperparameter; schedulers automate its tuning.
  • Saddle PointsIn high dimensions, most critical points are saddle points, not local minima. GD with noise easily escapes them.
  • Batch Size and GeneralisationLarge batches often find sharp minima that generalise poorly. Smaller batches add gradient noise that regularises. Rule of thumb: scale LR linearly with batch size.

Live: 3D/Contour Loss Surface Descent Race

Watch SGD (cyan), Momentum (pink), and Adam (amber) traverse a non-convex Himmelblau loss surface from identical starting conditions.

■ SGD ■ Momentum ■ Adam

Quick answers

What is Gradient Descent?

Gradient Descent is an iterative optimisation algorithm that updates model parameters in the direction of steepest loss decrease. Stochastic and mini-batch variants sample subsets of data per step for computational efficiency, introducing useful noise that aids generalisation.

Why does Adam optimiser converge faster than vanilla SGD?

Adam maintains per-parameter adaptive learning rates by tracking a running mean and uncentred variance of past gradients. Parameters with consistently large gradients get smaller effective rates; sparse parameters get larger rates. This makes Adam robust to gradient scale differences across layers.