Token
What is it?
A token is a chunk of text — often a word or part of a word — that language models read and generate one at a time.
Explain like I'm 5
Why was it created?
Models need text in consistent, manageable units. Tokenization was created to break language into pieces a model can process and predict.
Where is it used?
- Measuring LLM input/output size
- Pricing AI API usage
- Enforcing context limits
- Counting prompt length
Why should developers care?
Tokens determine model cost, speed, and limits, so anyone building with LLMs budgets and reasons in tokens.
How does it work?
A tokenizer splits text into tokens using learned rules, turning each into a number the model understands. The model predicts the next token repeatedly to generate text, then the tokens are turned back into readable words.
Real-world example
The word 'unhappiness' might be split into a few tokens like 'un', 'happi', 'ness', each processed separately by the model.
Common use cases
- Estimating API cost
- Staying within context limits
- Measuring prompt size
- Comparing model efficiency
Advantages
- Consistent processing units
- Handles unknown words via fragments
- Enables clear cost/length measurement
- Language-agnostic
Disadvantages
- Token counts aren't intuitive
- Differ between models
- Some languages use more tokens per word
- Easy to underestimate length
When should you use it?
Whenever you need to reason about LLM length, cost, or limits.
When should you avoid it?
Not something to avoid — it's an underlying unit, useful to understand rather than opt out of.
Alternatives
Related terms
Interview questions
Beginner
- What is a token?
- Is a token always a whole word?
Intermediate
- Why do models use tokens instead of words?
- Why does token count affect cost?
Senior
- How does tokenization affect non-English text cost?
- Why might the same text differ in token count across models?
Common misconceptions
- "One token equals one word" — a token is often a word fragment; long or rare words become several tokens.
- "Token counts are the same everywhere" — different models tokenize differently.
Fun facts
- A rough rule of thumb is that a token is about four characters of English text.
- Both your prompt and the model's reply consume tokens.
Timeline
- 2010s — Subword tokenization becomes standard for language models
Learning resources
Quick summary
A token is a chunk of text (often a word fragment) that models process one at a time, and the unit that governs LLM cost and limits.
Cheat sheet
- Text chunk, ~a word or fragment
- Models read/write token by token
- Drives cost and context limits
- Varies by model