SK
Back to AI & ML TrackSearch Modules
Modules
1. What is a Neuron?
2. Neural Networks
3. How Training Works
4. Word Embeddings
5. Attention Mechanism
6. The Transformer
7. How LLMs Work
8. RAG Pipelines
9. Building AI Agents
10. Prompt Engineering
11. Linear Regression
12. Logistic Regression
13. Decision Trees
14. Random Forest
15. K-Means Clustering
16. Cosine Similarity
17. Types of AI & AI Agents
Share Guide
AI & MLDeep
Intermediate

The Transformer

14 min read
Learn
Deep Reading
Estimated 14 mins
Prereq
Intermediate
Basic ML concepts helpful
Interactive
Static Playbook
Static guide & reference tables

The paper that changed AI

In 2017, Google researchers published "Attention Is All You Need." The title was the whole argument: you could build a state-of-the-art sequence model using nothing but attention mechanisms, with no recurrence and no convolutions. The resulting architecture — the transformer — became the foundation for BERT, GPT, T5, LLaMA, Claude, and essentially every major language model that exists today.

The key insight: RNNs process sequences one token at a time, which is slow and makes it hard to connect information from far apart in a sequence. Transformers process every token in the sequence simultaneously, and the attention mechanism lets every token directly attend to every other token — regardless of distance.

The problem with recurrence

Before transformers, the dominant approach for sequence tasks was the recurrent neural network (RNN) and its variants (LSTM, GRU). These worked by passing a hidden state from one token to the next — essentially a fixed-size summary of everything seen so far.

The problems were fundamental: sequential processing made parallelism impossible (you could not train on GPUs efficiently), and information from early in the sequence had to travel through hundreds of steps to influence the output, often getting diluted or lost. Transformers solve both problems at once.

Note

Because transformers process all tokens simultaneously, training can be massively parallelised across thousands of GPU cores. This is the physical reason transformers enabled the era of large language models — not just better architecture, but an architecture that could actually exploit modern hardware.

Positional encoding: telling the model where things are

Attention is permutation-invariant — it doesn't care what order the tokens arrive in. The word "dog bites man" and "man bites dog" would produce identical attention matrices if we didn't inject position information. Positional encoding solves this by adding a position-dependent signal to each token's embedding before it enters the transformer.

The original paper used sine and cosine functions at different frequencies for each dimension of the embedding. Modern models often use learned positional embeddings instead. Either way, the model can now distinguish token 1 from token 50 even though the attention mechanism itself is order-agnostic.

python

Multi-head attention

You've seen how single-head attention works: compute queries, keys, and values, then score every token against every other token, and produce a weighted mix of values.

Multi-head attention runs this process several times in parallel, with different learned weight matrices — different "heads." Each head attends to different aspects of the input. One head might track grammatical relationships, another might track coreference (which pronoun refers to which noun), another might track semantic similarity.

The outputs from all heads are concatenated and projected back to the model dimension. This gives the model much richer representational power than a single attention computation.

python

The full transformer block

A transformer is built by stacking identical blocks. Each block has two sub-layers:

1. Multi-head self-attention — every token attends to every other token
2. Feed-forward network — a two-layer MLP applied independently to each position

Both sub-layers are wrapped with a residual connection (the input is added to the output) and layer normalisation. Residual connections are crucial: they give gradients a direct path back through the network during training, solving the vanishing gradient problem and enabling very deep stacking.

GPT-3 stacks 96 of these blocks. Each block refines the representation a little more, with later blocks typically encoding higher-level semantic information.

Note

BERT uses only the encoder — it reads the full sequence bidirectionally (every token can attend to every other token). This makes it great for classification and understanding tasks. GPT uses only the decoder — tokens can only attend to previous tokens (causal masking), which makes it great for generation. T5 and the original transformer use both: an encoder processes the input, a decoder generates the output while attending to the encoder's representations.

Why transformers beat everything before them

Three compounding advantages:

Expressivity — multi-head attention can model arbitrarily complex relationships between any two tokens in the sequence, regardless of distance. RNNs struggle with long-range dependencies; transformers handle them natively.

Scalability — the architecture scales gracefully. Add more layers, more heads, larger embeddings: performance keeps improving with compute. This is the empirical finding behind scaling laws — transformer performance improves predictably as you scale model size and training data.

Parallelism — the entire sequence is processed simultaneously, making training on modern GPU clusters practical in a way recurrent models never were.

What's next

The transformer is the foundation, but the LLM is what you get when you train one on a massive corpus with next-token prediction. The next lesson — How LLMs Work — covers pre-training, fine-tuning, RLHF, and what's actually happening when a model generates text.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need custom AI or MarTech setup? Let's build together.

Let's talk →