Transformer
What is it?
A transformer is a neural network design that processes sequences (like text) by weighing how much each part should pay attention to every other part.
Explain like I'm 5
Why was it created?
Older sequence models processed words one at a time and struggled with long-range context. Transformers were created to handle whole sequences in parallel using attention.
Where is it used?
- Large language models
- Translation and summarization
- Image and audio models
- Code generation
Why should developers care?
Transformers are the architecture behind today's large language models, so they underpin most modern AI you'll build with.
How does it work?
A transformer uses a mechanism called attention: for each element in a sequence, it computes how relevant every other element is and blends them accordingly. Because this happens in parallel across the sequence, it trains efficiently on large data.
Real-world example
When a model translates a sentence, attention lets it link each output word to the most relevant input words, even across long distances.
Common use cases
- Language understanding and generation
- Translation and summarization
- Powering LLMs
- Multimodal models (text, images)
Advantages
- Captures long-range context
- Processes sequences in parallel
- Scales well with data and compute
- Versatile across modalities
Disadvantages
- Expensive to train
- Compute grows with sequence length
- Needs large datasets
- Hard to interpret
When should you use it?
For sequence and language tasks where context and scale matter.
When should you avoid it?
For tiny datasets or simple tasks where lighter models suffice.
Alternatives
Related terms
Interview questions
Beginner
- What kind of data do transformers handle well?
- Why are transformers important for LLMs?
Intermediate
- What is the attention mechanism?
- Why can transformers process sequences in parallel?
Senior
- Why does compute scale with sequence length?
- What trade-offs come with very long context windows?
Common misconceptions
- "Transformers are a kind of robot" — it's a neural network architecture, not a physical machine.
- "Attention means the model truly understands" — it's a weighting mechanism over the input, not comprehension.
Fun facts
- The architecture was introduced in a 2017 paper titled 'Attention Is All You Need'.
- The 'GPT' in many model names stands for Generative Pre-trained Transformer.
Timeline
- 2017 — The transformer architecture is introduced
Learning resources
Quick summary
A transformer is a neural architecture that uses attention to weigh relationships across a whole sequence, powering modern language models.
Cheat sheet
- Sequence model using attention
- Weighs every part against every other
- Processes in parallel; scales well
- Foundation of LLMs