Curriculum

Topic 04 · Language & Text

Natural Language Processing (NLP)

NLP turns raw text into machine-readable representations. Tokenisation, embeddings, and language models are the pipeline that lets machines read, write and reason about language.

The idea

Text is discrete and variable-length—unlike images, you cannot feed raw characters into a matrix multiply. NLP pipelines solve this. Tokenisation splits text into units (words, subwords, or bytes). Embeddings map tokens to dense vectors in a continuous space where similar words are geometrically close.

Modern tokenisers use Byte-Pair Encoding (BPE) or SentencePiece: start with individual characters, greedily merge the most frequent pair until reaching a target vocabulary size. GPT-4 uses ~100k BPE tokens; the same tokeniser encodes any language.

sim(A,B) = (A·B) / (||A||·||B||)

Cosine similarity between embedding vectors—the standard distance metric in NLP retrieval.

What to hold onto

  • Word2VecPredict a word from its context (CBOW) or context from a word (Skip-gram). Produces embeddings where "king − man + woman ≈ queen".
  • Stemming vs. LemmatisationStemming chops suffixes aggressively (running→run, ran→ran). Lemmatisation uses morphological analysis (ran→run). Lemmatisation is slower but more accurate.
  • Stop WordsHigh-frequency function words (the, and, is) often add noise in retrieval. Removing them can improve precision but hurts semantic understanding in modern transformers.

Live: Byte-Pair Encoding (BPE) & Word Vector Math

Live BPE subword segmentation with vocabulary indices, plus vector geometry $\vec{v}(\text{King}) - \vec{v}(\text{Man}) + \vec{v}(\text{Woman}) \approx \vec{v}(\text{Queen})$.

Quick answers

What is Natural Language Processing?

NLP is the field of AI that enables computers to understand, interpret, and generate human language. Core tasks include tokenisation (splitting text into units), embedding (converting tokens to vectors), and language modelling (predicting or generating text). Modern NLP is dominated by Transformer-based models pre-trained on large corpora.

What is tokenisation?

Tokenisation splits raw text into discrete units called tokens—words, subwords, or characters. Modern systems like GPT use Byte-Pair Encoding, a subword method that balances vocabulary size against character coverage, handling any language and rare words without an "unknown" token.