Attention Mechanism
What is it?
Attention is the mechanism that lets a model weigh which other words in the input matter most for understanding each word, no matter how far apart they are.
Explain like I'm 5
Why was it created?
Older sequence models processed text word by word and struggled to connect distant words. Attention was introduced so a model could directly relate any word to any other, in parallel.
Where is it used?
- Transformers and LLMs
- Machine translation
- Vision and multimodal models
- Any long-range sequence task
Why should developers care?
Attention is the core idea behind transformers, which power today's LLMs — understanding it demystifies how modern models actually work.
How does it work?
For each token the model builds a query, and every token offers a key and a value. Comparing the query to all keys yields attention weights; the token's new representation is a weighted blend of the values. 'Self-attention' does this within one sequence, letting every word attend to every other.
Real-world example
Translating a sentence, the model attends strongly to the subject noun when choosing the correct verb agreement, even across a long clause.
Common use cases
- Resolving references (what 'it' means)
- Long-range dependencies
- Aligning source and target in translation
- Relating image patches or modalities
Advantages
- Connects distant tokens directly
- Highly parallelizable
- Flexible across text, vision, audio
- Weights are somewhat interpretable
Disadvantages
- Cost grows with sequence length (quadratic)
- Memory-heavy for long inputs
- Weights don't fully explain reasoning
- Needs lots of data to train well
When should you use it?
It's built into transformer models; you rely on it whenever you use an LLM or modern sequence model.
When should you avoid it?
For tiny, purely local problems, simpler models may be cheaper — but you rarely choose to 'avoid attention' directly.
Alternatives
Related terms
Interview questions
Beginner
- What does attention help a model do?
- What is self-attention?
Intermediate
- What are queries, keys, and values?
- Why is attention cost quadratic in sequence length?
Senior
- How does multi-head attention help?
- What approaches reduce attention's quadratic cost for long contexts?
Common misconceptions
- "Attention weights are the model's explanation" — they're informative but not a full account of reasoning.
- "Attention is unique to language" — it's used in vision and multimodal models too.
Fun facts
- The 2017 paper that scaled it up was titled 'Attention Is All You Need'.
- Multi-head attention runs several attention computations in parallel and combines them.
Timeline
- 2014 — Attention introduced for machine translation
- 2017 — Transformers make self-attention the core building block
Learning resources
Quick summary
Attention lets a model weigh how much every token matters to every other, enabling long-range understanding and powering transformers.
Cheat sheet
- Weighs which tokens matter to each token
- Query–key–value blend
- Core of transformers/LLMs
- Cost grows quadratically with length