Curriculum

Deep Learning · Non-Linearities

Activation Functions

Activation functions inject non-linearity into networks. Without them, stacked layers collapse to a single linear map. ReLU dominates hidden layers; Softmax converts logits to class probabilities.

The idea

Every neuron needs to break linearity. If every layer is f(x) = Wx + b, the composition is still affine. A non-linearity between each layer lets the network represent curved decision boundaries.

Sigmoid: σ(x) = 1/(1+e⁻ˣ). Outputs ∈ (0,1). Saturates at large |x|—gradients vanish. Historically used in hidden layers, now mainly for binary output neurons.

Tanh: (eˣ−e⁻ˣ)/(eˣ+e⁻ˣ). Outputs ∈ (−1,1). Zero-centred (better than sigmoid) but still saturates. Preferred over sigmoid in RNNs.

ReLU: max(0,x). No saturation for positive inputs. Computationally cheap. Can "die" (zero gradient for all negative inputs). Dominant in CNNs and feedforward nets.

Softmax: softmax(xᵢ) = eˣⁱ/Σⱼeˣʲ. Converts a vector of logits to a probability distribution summing to 1. Standard final layer for multi-class classification.

ReLU(x) = max(0, x) Softmax(xᵢ) = eˣⁱ / Σⱼ eˣʲ

The two most important activations: hidden layers use ReLU; multi-class output layers use Softmax.

What to hold onto

  • Leaky ReLUmax(αx, x) with α≈0.01 prevents dying neurons by allowing a small gradient for negative inputs.
  • GELUGaussian Error Linear Unit, used in BERT and GPT. Smooth approximation of ReLU, slightly better empirically on Transformers.
  • Swish / SiLUx·σ(x). Self-gated, smooth, non-monotonic. Used in EfficientNet and some LLMs. Slightly outperforms ReLU on many tasks.

Live: Activation Function & Derivative Curve Plotter

Select an activation function and drag the input $x$ to view both $f(x)$ (cyan) and its derivative $f'(x)$ (pink).

f(x): 0.50 f'(x): 0.25

Quick answers

What are activation functions in neural networks?

Activation functions introduce non-linearity between layers of a neural network. Without them, stacking layers reduces to a single linear transformation. They determine the output range of a neuron, influence gradient flow during backpropagation, and are critical to a network's expressive power.

Why is ReLU preferred over Sigmoid in hidden layers?

ReLU does not saturate for positive inputs, so its gradient is exactly 1 there—no vanishing gradient. Sigmoid saturates at both extremes, driving gradients to near zero and slowing deep network training. ReLU is also computationally trivial: a single threshold operation.