Tokenizer
What is it?
A tokenizer is the tool that splits text into tokens — the small chunks (word pieces) a language model actually reads and generates.
Explain like I'm 5
Why was it created?
Models work with numbers, not raw text, and whole words are too many to list. Tokenizers were created to break text into a manageable, reusable vocabulary of sub-word pieces.
Where is it used?
- Every LLM prompt and response
- Counting cost and context usage
- Preparing training data
- Multilingual and code handling
Why should developers care?
Tokens determine cost, context limits, and quirks like why models miscount letters — so understanding tokenization explains a lot of real LLM behavior.
How does it work?
The tokenizer maps text to token IDs using a learned vocabulary of sub-word units (common methods: BPE, WordPiece). Frequent words become single tokens; rare or novel words split into pieces. The model reads these IDs and outputs token IDs, which are decoded back to text.
Real-world example
'unbelievable' might become 'un', 'believ', 'able' — three tokens — while 'cat' is one; this is why your bill and context usage depend on tokens, not words.
Common use cases
- Estimating token cost
- Fitting text within a context window
- Chunking documents for retrieval
- Debugging odd model behavior
Advantages
- Compact, reusable vocabulary
- Handles unknown words via sub-words
- Works across languages and code
- Enables numeric input for models
Disadvantages
- Token boundaries can be unintuitive
- Some languages use far more tokens
- Causes character-level blind spots
- Different models tokenize differently
When should you use it?
Whenever you need to reason about cost, context limits, or how a model chunks and reads your text.
When should you avoid it?
You rarely avoid it — but you don't need to hand-tune tokenization for typical app development.
Alternatives
Related terms
Interview questions
Beginner
- What does a tokenizer produce from text?
- Why isn't one token always one word?
Intermediate
- Why can an LLM struggle to count letters in a word?
- How does sub-word tokenization handle unseen words?
Senior
- How does byte-pair encoding build its vocabulary?
- Why can the same text cost more tokens in one language than another?
Common misconceptions
- "One token equals one word" — tokens are often word fragments.
- "Tokenization doesn't affect quality" — it shapes cost, limits, and some failure modes.
Fun facts
- A rough rule of thumb: ~4 characters per token in English.
- Poor tokenization is a big reason models fumble tasks like counting characters.
Timeline
- 2016 — Byte-pair encoding popularized for neural translation
- 2020s — Sub-word tokenizers standard across LLMs
Learning resources
Quick summary
A tokenizer splits text into sub-word tokens the model reads and writes, which drives cost, context limits, and some model quirks.
Cheat sheet
- Splits text into tokens (word pieces)
- ~4 chars/token in English
- Drives cost & context usage
- Explains character-counting errors