Curriculum

Edge AI · Efficient Inference

Small Language Models (SLM)

SLMs deliver near-LLM capability with a fraction of the parameters, enabling local execution on edge devices. Knowledge distillation and domain specialisation are the key techniques.

The idea

Large models (>70B parameters) require multi-GPU inference, incurring latency and cost. SLMs (1B–14B parameters) run on a single GPU or even CPU with quantisation. Flagship SLMs: Phi-3 (Microsoft), Gemma 2 (Google), Llama 3.2 (Meta), Mistral 7B.

Knowledge Distillation: train a small "student" model to mimic the output distribution (soft labels) of a large "teacher" model, not just the hard labels. The student learns the teacher's uncertainty, yielding better generalisation than training on hard labels alone. Distillation loss: L = α·CE(student, hard) + (1−α)·KL(student, teacher).

Domain Specialisation: continue pre-training on domain-specific text (medical, legal, code) then fine-tune. A 7B model specialised in clinical notes can outperform GPT-4 on medical tasks at 1/100th the inference cost.

L = α·CE(ŷ_student, y) + (1−α)·KL(ŷ_student || ŷ_teacher)

Knowledge Distillation loss: weighted sum of hard-label cross-entropy and soft-label KL divergence.

What to hold onto

  • Parameter EfficiencyPhi-3-mini (3.8B) matches Llama-3-70B on many benchmarks through data curation: training on textbook-quality synthetic data versus raw web scrapes.
  • On-Device InferenceApple ANE, Qualcomm AI Engine, Google Tensor G4 run 7B models at 50+ tokens/second on consumer phones with 4-bit quantisation.
  • Speculative DecodingUse a tiny draft model to propose k tokens; the large model verifies them in one forward pass. 2–4× throughput improvement with identical output quality.

Live: Knowledge Distillation Soft-Target Transfer

Adjust distillation temperature $T$ to observe softened teacher logits transferring dark knowledge to student.

Teacher (70B) → Student (3B)

Quick answers

What is a Small Language Model (SLM)?

SLMs are language models with 1B–14B parameters designed for efficient inference on edge devices and consumer hardware. They achieve near-LLM performance through knowledge distillation, high-quality curated training data, and domain specialisation, enabling local execution without cloud API calls.

What is knowledge distillation?

Knowledge distillation trains a small student model to mimic a large teacher model's output probability distributions (soft labels), not just the hard ground truth labels. The soft labels convey the teacher's confidence and inter-class relationships, enabling the student to achieve higher accuracy than training on hard labels alone.