Curriculum

Generative AI · Embedding Storage

Vector Databases

Vector databases store high-dimensional embeddings and enable sub-linear approximate nearest neighbour search. They are the retrieval backbone of RAG systems and semantic search engines.

The idea

Traditional databases index values with B-Trees, optimised for exact lookups. Vector databases index dense floating-point vectors for similarity search—finding the k vectors most similar to a query vector by cosine or Euclidean distance.

HNSW (Hierarchical Navigable Small World) is the dominant indexing algorithm. It builds a multi-layer proximity graph: top layers have long-range connections for fast traversal; bottom layers have short-range connections for precision. Search starts at the top and greedily descends, achieving O(log n) average query time.

Popular systems: Pinecone (managed cloud), Weaviate (open-source), Qdrant (Rust-native), Milvus (distributed), pgvector (PostgreSQL extension). Each trades off index build time, query latency, update performance, and cost.

d(a,b) = 1 − (a·b)/(||a||·||b||)

Cosine distance—1 minus cosine similarity. The primary metric for embedding similarity search.

What to hold onto

  • ANN vs. Exact SearchApproximate Nearest Neighbour (ANN) accepts a small accuracy loss for massive speed gains. HNSW achieves >95% recall at 10× the speed of brute-force for million-scale datasets.
  • QuantisationProduct Quantisation (PQ) compresses each vector by representing it as a sequence of sub-vector codebook IDs. Reduces memory 4–32×. IVF+PQ is a common hybrid.
  • Metadata FilteringMost vector DBs support hybrid query: filter by metadata (date, category, user_id) and then run similarity search within matching documents. Critical for multi-tenant RAG.

Live: HNSW Vector Proximity Search

Hierarchical Navigable Small World (HNSW) multi-layer graph navigation with fast greedy convergence.

Click to query

Quick answers

What is a Vector Database?

A vector database stores high-dimensional embeddings and provides efficient approximate nearest-neighbour search. It enables semantic retrieval—finding the most semantically similar documents to a query by comparing embedding vectors rather than exact keywords. Essential infrastructure for RAG, recommendation, and semantic search systems.

What is HNSW indexing?

Hierarchical Navigable Small World is a graph-based ANN algorithm that builds a multi-layer proximity graph. Queries start at coarse upper layers for rapid traversal and descend to precise lower layers. HNSW achieves logarithmic average search time and high recall, making it the industry standard for large-scale embedding search.