Token
The small piece of text an AI reads and writes — usually a chunk of a word, not a whole word.
When not to use it
- Estimating length for a human audience. Readers care about words and pages; tokens are a machine unit. Quoting "4,000 tokens" to a client tells them nothing.
- Optimising prompts before you have a cost problem. Shaving tokens is the last 10% of a cost fix — switching to a smaller model or caching repeated context usually saves far more.
- As a proxy for difficulty or quality. A short prompt isn't a good prompt, and a long answer isn't a thorough one.
Reach for something else instead
- Character counts are fine when you just need a rough guard against absurd inputs, and they're free to compute.
- Word counts are the right unit for anything a person reads or approves.
- The model's own tokenizer is the only reliable answer when the number actually matters for cost or limits. Every rule of thumb, including "¾ of a word," breaks on code, numbers, and non-English text.
Sources & further reading
- Sennrich, Haddow & Birch (2016), Neural Machine Translation of Rare Words with Subword Units — the paper that made byte-pair encoding standard in NLP.
- Gage (1994), A New Algorithm for Data Compression — BPE's origin, as a compression scheme, two decades before anyone applied it to language models.
- Kudo & Richardson (2018), SentencePiece — the language-independent tokenizer used by many non-GPT models.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming the ¾-of-a-word rule holds everywhere. It's an English-prose average. Japanese, Arabic, JSON, and code routinely cost two to three times more per visible character.
- Forgetting output tokens count too. Context limits and bills cover prompt plus answer, which is why long responses get truncated at what feels like an arbitrary point.
- Blaming the model for bad arithmetic when the tokenizer is the culprit. Numbers split in strange places, so digits the model needs to compare may not be in the same token at all.
At a glance
FieldLanguage & LLMs
Core ideasubword chunks of text
Rule of thumb~¾ of a word
Why it mattersthe unit of cost, context, and limits
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4