Tokenization
Cutting text into the pieces a model actually reads — the least glamorous step in the stack, and the cause of a surprising share of its stupidest failures.
When not to use it
- (You can't avoid it. The question is when it defeats you.)*
- For character-level tasks. Counting letters, reversing strings, rhyme, syllables. The model can't see inside the token. Use code.
- For arithmetic you care about. Numbers chunk inconsistently. Use a calculator tool.
- When estimating cost for non-English. The English rules of thumb are wrong, sometimes by 10×.
- When constructing prompts programmatically without counting.
"hello"≠" hello".
Reach for something else instead
- Byte-level models — no vocabulary, no asymmetry, longer sequences.
- Character-level — same trade, more extreme.
- A tool call — for anything character- or number-precise, don't ask the model to see what it can't.
- A language-appropriate tokenizer — if you're building, don't inherit an English-fit vocabulary.
Sources & further reading
- Sennrich, Haddow & Birch (2016), Neural Machine Translation of Rare Words with Subword Units — BPE repurposed from compression to NLP.
- Kudo & Richardson (2018), SentencePiece: A simple and language independent subword tokenizer — no whitespace assumption, which matters outside European languages.
- Petrov et al. (2023), Language Model Tokenizers Introduce Unfairness Between Languages — order-of-magnitude cost differences for identical content.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Blaming the model's intelligence for character-level failures. It never saw the characters.
- Using 4-chars-per-token for code, JSON or non-English. It's wrong for all three.
- Assuming token pricing is language-neutral. It isn't, by a lot.
- Missing that leading spaces change tokens, then debugging the wrong thing.
At a glance
FieldLanguage & LLMs
What it istext cut into fixed-vocabulary chunks
Origina 1994 compression algorithm, repurposed in 2016
Why models can't count lettersthey never saw letters
The fairness problemsome languages cost 10× more for identical content
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Tokenization vs. byte-level — one compresses and buys you shorter sequences at the cost of character blindness and language unfairness; the other reads raw bytes and pays in length.