Curriculum

ML Domain · Policy Optimisation

Reinforcement Learning

RL trains agents to make sequential decisions by rewarding desired outcomes. The agent learns a policy mapping states to actions by maximising cumulative reward over time.

The idea

Reinforcement Learning frames the learning problem as an agent interacting with an environment. At each step the agent observes a state, takes an action, and receives a scalar reward. The objective is to learn a policy—a mapping from states to actions—that maximises total discounted future reward.

Markov Decision Processes (MDPs) provide the mathematical framework. An MDP is defined by states S, actions A, transition dynamics P(s'|s,a), and a reward function R(s,a). The Markov property says the future is independent of the past given the present state.

Q-Learning learns a value function Q(s,a)—the expected total reward from taking action a in state s and following the optimal policy thereafter. Deep Q-Networks (DQN) parameterise Q with a neural network, enabling RL in high-dimensional spaces like Atari games.

Q(s,a) ← Q(s,a) + α[r + γ·maxₐ'Q(s',a') − Q(s,a)]

Bellman update for Q-Learning. α is the learning rate, γ is the discount factor.

What to hold onto

  • Exploration vs. Exploitationε-greedy: explore randomly with probability ε, exploit the best known action otherwise. Annealing ε over time is standard practice.
  • Policy GradientsREINFORCE and PPO directly optimise the policy by ascending the gradient of expected reward. Better for continuous action spaces.
  • Reward ShapingSparse rewards are hard to learn from. Shaped rewards provide denser signal but must be carefully designed to avoid gaming.

Live: Q-Learning GridWorld Agent

Watch an agent learn optimal state-action values $Q(s, a)$ through trial-and-error rewards (+100 goal, -50 trap, -1 step).

Episode: 0 | Steps: 0 | Total Reward: 0 Arrows: Greedy Policy π(s)

Quick answers

What is Reinforcement Learning?

Reinforcement Learning is a machine learning paradigm where an agent learns to make decisions by interacting with an environment, receiving rewards for desired behaviours and penalties for undesired ones. It is the basis for game-playing AI (AlphaGo) and robotics.

What is a Markov Decision Process?

An MDP models sequential decision making with states, actions, transition probabilities, and a reward function. The Markov property ensures the future depends only on the current state, not history, making the mathematics tractable.