Subword Tokenization
The compromise that lets a fixed vocabulary cover any word: split rare words into frequent pieces, and accept that the model never sees letters.
Reviewed July 15, 2026Moving
When not to use it
- As an explanation for every model error. Most mistakes are not tokenization artefacts.
- As a fixed property of a model family. Different models in the same family may tokenize differently.
Reach for something else instead
- Character-level models, which remove the artefacts and lengthen sequences considerably.
- Byte-level models, which do the same with guaranteed coverage of any input.
- Word-level, effectively obsolete because of vocabulary size and unseen words.
Read more on the blog
- Why AI is bad at math, and which failures are permanentChange how a number is split into tokens and accuracy shifts substantially, which means the arithmetic was following the tokenizer rather than a method. Some failures improve with better models. These do not.
- What linguistics predicts about where AI failsLinguistic features stopped improving models around 2019, because models learn the structure themselves. The categories survived anyway, as the best available map of where these systems break.
- Why AI works worse in your languageThe gap between English and everything else is not about linguistic difficulty. It is about training data share, tokenizer fitting, and where instruction-tuning stopped, and only one of those three is expensive to fix.
Further reading
- Sennrich et al. (2016), Neural Machine Translation of Rare Words with Subword Units — introduced BPE for NLP.
- Kudo (2018), Subword Regularization — the Unigram alternative and why sampling segmentations helps.
- Ahia et al. (2023), Do All Languages Cost the Same? — the cross-script cost consequences.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Estimating cost in words. Tokens and words diverge sharply outside plain English.
- Assuming a token is a syllable or a word piece with linguistic meaning. Merges are statistical, not morphological.
- Expecting reliable character-level manipulation, which the representation does not support well.
At a glance
FieldLanguage & LLMs
Dominant methodbyte-pair encoding
Typical vocabulary32k to 200k
Explainsletter counting failures, cost variation by script
Frozen atpretraining
DifficultyHands-on
Where this sits
2 concepts come first. Understanding it opens up 2 more.
2Levelsteps in
2Needs firstconcepts
2Opens up1% of 310
2Areascrosses fields
Learn these firstTokenization
Computed from the prerequisite graph, not assigned. How this works