Tokenization
Cutting text into the pieces a model actually reads — the least glamorous step in the stack, and the cause of a surprising share of its stupidest failures.
Reviewed July 12, 2026Stable
When not to use it
- (You can't avoid it. The question is when it defeats you.)*
- For character-level tasks. Counting letters, reversing strings, rhyme, syllables. The model can't see inside the token. Use code.
- For arithmetic you care about. Numbers chunk inconsistently. Use a calculator tool.
- When estimating cost for non-English. The English rules of thumb are wrong, sometimes by 10×.
- When constructing prompts programmatically without counting.
"hello"≠" hello".
Reach for something else instead
- Byte-level models — no vocabulary, no asymmetry, longer sequences.
- Character-level — same trade, more extreme.
- A tool call — for anything character- or number-precise, don't ask the model to see what it can't.
- A language-appropriate tokenizer — if you're building, don't inherit an English-fit vocabulary.
Read more on the blog
- Why AI is bad at math, and which failures are permanentChange how a number is split into tokens and accuracy shifts substantially, which means the arithmetic was following the tokenizer rather than a method. Some failures improve with better models. These do not.
- Why AI thinks hot and cold mean the same thingWord embeddings score "accept" and "reject" at 0.73 similarity. The reason is the idea embeddings are built on, and it explains why cosine similarity is a weaker signal than most systems treat it as.
- How a sentence becomes an answer: an LLM end to endMost explanations of large language models cover one piece, attention, or tokens, or sampling, in isolation. This follows a single sentence all the way through the machine, from the moment you hit enter to the words that come back, so the pieces finally connect.
- What is a token? How AI reads text in chunksAsk a model how many r's are in "strawberry" and it often says two. The answer is three. It is not bad at counting; it never sees the letters. Text is chopped into tokens before the model reads it, and that single fact explains a surprising amount about how AI behaves, what it costs, and what it fails at.
Tokenization splits text into the pieces a model actually reads.
A model doesn't see words — it sees tokens, sub-word chunks from a fixed vocabulary. Common words are one token; rarer ones split into pieces ('tokenization' → 'token' + 'ization'). This is why models count length in tokens, not characters, and why unusual words cost more of the context budget.
Further reading
- Sennrich, Haddow & Birch (2016), Neural Machine Translation of Rare Words with Subword Units — BPE repurposed from compression to NLP.
- Kudo & Richardson (2018), SentencePiece: A simple and language independent subword tokenizer — no whitespace assumption, which matters outside European languages.
- Petrov et al. (2023), Language Model Tokenizers Introduce Unfairness Between Languages — order-of-magnitude cost differences for identical content.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Blaming the model's intelligence for character-level failures. It never saw the characters.
- Using 4-chars-per-token for code, JSON or non-English. It's wrong for all three.
- Assuming token pricing is language-neutral. It isn't, by a lot.
- Missing that leading spaces change tokens, then debugging the wrong thing.
At a glance
FieldLanguage & LLMs
What it istext cut into fixed-vocabulary chunks
Origina 1994 compression algorithm, repurposed in 2016
Why models can't count lettersthey never saw letters
The fairness problemsome languages cost 10× more for identical content
DifficultyBeginner
Often compared with
Tokenization vs. byte-level — one compresses and buys you shorter sequences at the cost of character blindness and language unfairness; the other reads raw bytes and pays in length.
Where this sits
1 concept come first. Understanding it opens up 70 more.
1Levelsteps in
1Needs firstconcepts
70Opens up23% of 310
7Areascrosses fields
Learn these firstNatural Language Processing
Computed from the prerequisite graph, not assigned. How this works