14. Beyond text: vision, speech and multimodal models

A transformer processes a sequence of vectors and doesn’t care where they came from. Cut an image into patches or audio into frames, project each piece to a vector, and the same architecture applies. Multimodal LLMs connect such encoders to a language model, so images and audio become tokens in the conversation.

Vision Transformer (ViT)

1. Split the image into fixed-size patches, e.g., 16 × 16 pixels; a 224 × 224 image gives 14 × 14 = 196 patches.
2. Flatten each patch (16 × 16 × 3 = 768 numbers) and project it linearly to d dimensions. This patch embedding plays the role of a token embedding.
3. Add position embeddings (2D position matters) and often a special [CLS] token whose output summarizes the image.
4. Run a standard transformer encoder.

CNNs build in assumptions - locality, translation equivariance - that help when data is scarce. ViTs have almost none, so they need more data or heavy augmentation, but they scale better and share all the tooling of language models.

CLIP: one space for images and text

CLIP (OpenAI, 2021) trains an image encoder and a text encoder together on hundreds of millions of image–caption pairs, with the contrastive loss from Part 13: matching pairs get similar embeddings, mismatched pairs dissimilar ones. The result is a shared embedding space. Zero-shot classification falls out of it: embed “a photo of a {dog, cat, car}” and pick the caption closest to the image. CLIP-style encoders power image search and act as the eyes of many multimodal LLMs.

Multimodal LLMs

The common design has three parts:

1. Vision encoder (often a CLIP-style ViT): turns an image into a grid of patch vectors.
2. Projector (a small MLP or a cross-attention resampler): maps those vectors into the LLM’s embedding space as “visual tokens”.
3. The LLM: reads visual tokens interleaved with text tokens and generates text.

Training usually aligns the projector first, with encoder and LLM frozen, on image–caption data, then instruction-tunes on visual question answering and documents. Systems implications: each image costs hundreds to thousands of tokens, and high-resolution images are often tiled into several crops, which affects cost and context; reading small text depends on resolution. Some newer models are natively multimodal, trained on mixed modalities from the start.

Speech

Whisper is a standard encoder–decoder. Audio becomes a log-mel spectrogram (a picture of frequencies over time), a small convolution stack downsamples it, the encoder processes the frames, and the decoder generates text tokens, including special tokens for language and timestamps. For speech generation and speech-to-speech models, neural audio codecs turn audio into discrete tokens a transformer can predict just like text.

Diffusion and diffusion transformers

Image and video generators mostly use diffusion: learn to remove noise step by step, starting from pure noise and denoising toward an image that matches a text prompt. The denoiser used to be a U-Net CNN; Diffusion Transformers (DiT) replace it with a transformer over patches of a compressed latent image, which scales like other transformers. Text conditioning usually enters through a text encoder and cross-attention or joint attention.

Questions you should now be able to answer

• How many tokens does a 448 × 448 image produce with 14 × 14 patches?
• Why does a ViT need more training data than a CNN of similar size?
• How does CLIP classify images zero-shot without a classification head?
• What does the projector in a multimodal LLM do, and why train it first?
• Why might a multimodal model misread small text in a screenshot, and what would you try?