Attention Mechanism
The core mechanic of modern Generative AI
Before transformers, neural networks processed language sequentially (one word at a time). This meant they struggled with long-range dependencies — words far apart in a sentence couldn't easily influence each other.
The Attention mechanism solved this by allowing the model to look at all tokens simultaneously and calculate how much they relate to each other. Instead of forcing information through a narrow bottleneck, the model dynamically focuses (attends) on the most relevant parts of the input sequence.
Self-Attention Heatmap
Type a sentence below and observe how the self-attention weights are calculated dynamically for each token pair. Darker purple squares indicate higher attention weights.
🧠 Attention Visualizer
InteractiveQuery, Key, and Value Vectors
To compute attention, the model projects each input token into three distinct vectors:
- Query ($Q$): what the token is looking for.
- Key ($K$): what information the token contains.
- Value ($V$): the actual content of the token.
The attention weight between token $i$ and token $j$ is determined by the dot product of the Query vector of $i$ and the Key vector of $j$. These scores are scaled, run through a softmax function to sum to 1.0, and then used to compute a weighted sum of the Value vectors.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.