Top-p (Nucleus) Sampling
What is it?
Top-p sampling limits a model's word choices to the smallest set of most-likely tokens whose probabilities add up to a threshold p, then picks from just those.
Explain like I'm 5
Why was it created?
Pure random sampling sometimes picks bizarre, low-probability words. Top-p was created to keep output varied but sensible by cutting off the unlikely tail dynamically.
Where is it used?
- Chat and completion APIs
- Creative generation
- Balancing variety and coherence
- Sampling configuration
Why should developers care?
Along with temperature, it's a core knob for controlling output quality, so knowing how they differ helps you tune models predictably.
How does it work?
The model ranks possible next tokens by probability and keeps adding them to a pool until their cumulative probability reaches p (e.g. 0.9). It then samples from that pool, ignoring the long tail of unlikely options. Lower p = safer, higher p = more variety.
Real-world example
With top-p 0.9, the model won't blurt an absurd word, but still has enough options to phrase things in fresh ways.
Common use cases
- Coherent but varied text
- Reducing nonsense tokens
- Tuning alongside temperature
- Creative writing with guardrails
Advantages
- Adapts the candidate set to the context
- Cuts the unlikely tail cleanly
- More robust than fixed top-k
- Simple to configure
Disadvantages
- Interacts with temperature confusingly
- Wrong value can be too flat or too rigid
- Not a quality guarantee
- Behaves differently across models
When should you use it?
When you want varied output without the occasional nonsense that pure sampling can produce.
When should you avoid it?
For fully deterministic tasks — use greedy/temperature 0 instead of tuning top-p.
Alternatives
Related terms
Interview questions
Beginner
- What does the p in top-p control?
- How is top-p different from top-k?
Intermediate
- Why is top-p called nucleus sampling?
- How do temperature and top-p combine?
Senior
- When would top-p adapt its candidate set more usefully than top-k?
- What happens at p=1 versus a very small p?
Common misconceptions
- "Top-p and top-k are the same" — top-k fixes a count; top-p fixes a probability mass.
- "Higher p is better" — it just widens choices; too high reintroduces nonsense.
Fun facts
- The kept pool is called the 'nucleus', hence 'nucleus sampling'.
- Many APIs let you set both temperature and top-p, though tuning one at a time is clearer.
Timeline
- 2019 — Nucleus sampling introduced ('The Curious Case of Neural Text Degeneration')
- 2020s — Standard sampling option in LLM APIs
Learning resources
Quick summary
Top-p sampling picks from the smallest set of top tokens covering probability mass p, keeping output varied but sensible.
Cheat sheet
- Keep top tokens summing to p
- Cuts the unlikely tail
- Pairs with temperature
- aka nucleus sampling