Subword Tokenization
The compromise that lets a fixed vocabulary cover any word: split rare words into frequent pieces, and accept that the model never sees letters.
Reviewed July 16, 2026Moving
When not to use it
- As an explanation for every model error. Most mistakes are not tokenization artefacts.
- As a fixed property of a model family. Different models in the same family may tokenize differently.
Reach for something else instead
- Character-level models, which remove the artefacts and lengthen sequences considerably.
- Byte-level models, which do the same with guaranteed coverage of any input.
- Word-level, effectively obsolete because of vocabulary size and unseen words.
Sources & further reading
- Sennrich et al. (2016), Neural Machine Translation of Rare Words with Subword Units — introduced BPE for NLP.
- Kudo (2018), Subword Regularization — the Unigram alternative and why sampling segmentations helps.
- Ahia et al. (2023), Do All Languages Cost the Same? — the cross-script cost consequences.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Estimating cost in words. Tokens and words diverge sharply outside plain English.
- Assuming a token is a syllable or a word piece with linguistic meaning. Merges are statistical, not morphological.
- Expecting reliable character-level manipulation, which the representation does not support well.
At a glance
FieldLanguage & LLMs
Dominant methodbyte-pair encoding
Typical vocabulary32k to 200k
Explainsletter counting failures, cost variation by script
Frozen atpretraining
DifficultyHands-on
Where this sits
2 concepts come first. Understanding it opens up 1 more.
2Levelsteps in
2Needs firstconcepts
1Opens up0% of 264
1Areastays here
Learn these firstTokenization
Computed from the prerequisite graph, not assigned. How this works