Curriculum

Generative AI · Cross-Modal Learning

Multimodal AI

Multimodal models process and generate across text, image, audio, and video simultaneously. Cross-modal alignment maps different modalities into a shared embedding space.

The idea

The challenge in multimodal AI is alignment: mapping semantically equivalent content from different modalities to nearby points in a shared embedding space. CLIP (Contrastive Language-Image Pretraining) does this by training an image encoder and a text encoder with contrastive loss—paired images and captions are pulled together; unpaired ones are pushed apart.

Modern architectures use a large pre-trained LLM as the "brain" and add modality-specific encoders (image, audio) connected via lightweight adapters (LLaVA, Flamingo, Gemini). The LLM learns to condition on tokens from any modality.

Visual Document Understanding (VDU) is an application where models read scanned PDFs with complex layouts—tables, figures, handwriting—by treating the document as an image rather than extracted text. Models like Donut and PaddleOCR-LM dominate.

L_CLIP = −Σᵢ log(exp(sᵢᵢ/τ) / Σⱼ exp(sᵢⱼ/τ))

CLIP contrastive loss over a batch. sᵢⱼ is the cosine similarity between image i and text j.

What to hold onto

  • Contrastive LossL = −log(exp(sim(iₙ,tₙ)/τ) / Σⱼexp(sim(iₙ,tⱼ)/τ)). Pulls matched pairs together, pushes mismatched pairs apart in embedding space.
  • Unified Token ApproachSome models (Chameleon, LlamaGen) unify all modalities as discrete tokens using a shared codebook. One transformer predicts all modalities seamlessly.
  • Cross-Modal RetrievalUse a shared embedding space to retrieve images given a text query (or vice versa). Powers Google Lens, Pinterest visual search.

Live: CLIP Cross-Modal Alignment Space

Joint embedding space aligning text prompts with image representations via contrastive cosine similarity.

Similarity: 0.884

Quick answers

What is Multimodal AI?

Multimodal AI systems process and generate across multiple data types—text, images, audio, video—simultaneously. They use cross-modal alignment techniques (like contrastive learning in CLIP) to map different modalities into a shared embedding space, enabling tasks like image captioning, visual question answering, and document understanding.

How does CLIP achieve cross-modal alignment?

CLIP trains an image encoder and a text encoder jointly with contrastive loss. Matched image-text pairs are pulled close in embedding space; mismatched pairs are pushed apart. This results in a shared space where text descriptions and images of the same concept have high cosine similarity.