Curriculum

Deep Learning · Learning Algorithms

Perceptron & Multi-Layer Perceptron

The Perceptron is the simplest trainable classifier. Stack multiple layers and you get an MLP capable of solving non-linear problems—including the XOR gate that defeated the single-layer version.

The idea

The Perceptron learning rule: if the output is wrong, adjust weights by adding or subtracting the input. Formally: w ← w + η·(y − ŷ)·x. Rosenblatt proved convergence for linearly separable data—but Minsky & Papert (1969) proved a single Perceptron cannot solve XOR, triggering the first AI winter.

The solution—already known in the late 1960s but popularised by Rumelhart, Hinton and Williams in 1986—was to add hidden layers and train with backpropagation. A Multi-Layer Perceptron (MLP) with at least one hidden layer and a non-linear activation can separate any convex region, and with sufficient depth, arbitrary geometries.

Geometrically, each hidden neuron defines a hyperplane. Each layer applies a learned coordinate transformation. Classification is asking whether the transformed point is in a particular region of the final space.

w ← w + η·(y − ŷ)·x

Perceptron update rule. η is the learning rate; y is the true label, ŷ the predicted label.

What to hold onto

  • Linear SeparabilityA single Perceptron can only separate classes if a hyperplane can divide them. XOR requires a curved boundary—impossible for one layer.
  • Hidden Layer IntuitionHidden neurons fold and stretch the input space so that what was non-linearly separable becomes linearly separable in the hidden representation.
  • Activation CriticalStep functions are not differentiable—backprop cannot work. Continuous activations (ReLU, sigmoid) are essential for MLP training.

Live: Perceptron Decision Boundary

Click = Cyan class · Shift+click = Pink class. Toggle MLP to see how multi-layer networks overcome the XOR linear separability bottleneck.

Single-layer: Linear Boundary

Quick answers

What is a Perceptron?

A Perceptron is the simplest binary classifier: it computes a weighted sum of inputs and fires if the result exceeds a threshold. It can only solve linearly separable problems. Adding hidden layers creates a Multi-Layer Perceptron capable of learning arbitrary decision boundaries.

Why can a single Perceptron not solve XOR?

XOR is not linearly separable—no single straight line (hyperplane) can divide the two output classes. A Multi-Layer Perceptron with at least one hidden layer applies non-linear transformations that reshape the input space, making XOR linearly separable in the learned representation.