Image Captioning
Two different kinds of model, stitched together
Image captioning generates a natural-language description of an image's content — not just a class label like 'dog,' but a full sentence like 'a golden retriever catching a frisbee in a park.' This requires combining two fundamentally different capabilities that neither computer vision nor natural language processing provides alone: understanding what's visually present in an image, and generating fluent, grammatically correct language describing it. The standard architectural pattern pairs a vision encoder, which turns an image into a compact feature representation, with a language decoder, which turns that representation into a word sequence.
The classic pipeline: CNN encoder, RNN decoder
Early and still-instructive captioning systems (like 'Show and Tell') used a CNN — often a classifier pretrained on ImageNet, with its final classification layer removed — as the vision encoder, producing a single fixed-size feature vector summarizing the whole image. That vector was then fed as the initial hidden state into a recurrent neural network (typically an LSTM), which generated the caption one word at a time, each word's prediction conditioned on the image feature and all previously generated words.
Note
Adding attention: looking at different regions for different words
The 'Show, Attend and Tell' approach fixed this by replacing the single global feature vector with a grid of regional feature vectors — one per spatial location in the CNN's late feature maps, instead of one for the whole image. At each decoding step, an attention mechanism computes a weighted combination of these regional vectors, with the weights determined by what word is currently being generated. Generating 'cat' lets the decoder put high attention weight on the image region actually containing the cat; generating 'lamp' shifts attention to a different region. This mirrors the same attention mechanism that later became central to transformers, applied here across spatial image regions instead of across sequence positions.
Modern approach: vision transformers and multimodal transformers
Current captioning systems typically replace the CNN+RNN combination with transformer components throughout. A Vision Transformer (ViT) or CNN backbone encodes the image into a sequence of patch or region embeddings; a transformer decoder (architecturally similar to GPT-style language models) generates the caption using self-attention over previously generated words and cross-attention over the image embeddings. Large multimodal models go further by training on vast paired image-text datasets at scale, enabling captions that go beyond listing objects to express relationships, context, and even some visual reasoning, and often sharing much of the same architecture used for other vision-language tasks like visual question answering.
Evolution of image captioning architectures
| Approach | Vision encoder | Language decoder | Key limitation |
|---|---|---|---|
| Show and Tell (CNN-RNN) | CNN, single global feature vector | LSTM, conditioned on fixed vector | No region-specific grounding; bottlenecked by one vector |
| Show, Attend and Tell | CNN, grid of regional feature vectors | LSTM with attention over regions | Attention is only over a coarse spatial grid |
| Transformer-based captioning | ViT or CNN, patch/region embeddings | Transformer decoder with cross-attention | Still needs large paired image-text training data |
Note
What's next
Image captioning sits at the intersection of computer vision and language generation, and the attention mechanism it popularized for grounding text in specific image regions became foundational to transformer-based vision-language models more broadly. Continue to transformers if you haven't already, to see the self-attention mechanism this field builds on in full generality.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.