Token
The small piece of text an AI reads and writes — usually a chunk of a word, not a whole word.
Reviewed July 10, 2026Stable
When not to use it
- Estimating length for a human audience. Readers care about words and pages; tokens are a machine unit. Quoting "4,000 tokens" to a client tells them nothing.
- Optimising prompts before you have a cost problem. Shaving tokens is the last 10% of a cost fix — switching to a smaller model or caching repeated context usually saves far more.
- As a proxy for difficulty or quality. A short prompt isn't a good prompt, and a long answer isn't a thorough one.
Reach for something else instead
- Character counts are fine when you just need a rough guard against absurd inputs, and they're free to compute.
- Word counts are the right unit for anything a person reads or approves.
- The model's own tokenizer is the only reliable answer when the number actually matters for cost or limits. Every rule of thumb, including "¾ of a word," breaks on code, numbers, and non-English text.
Read more on the blog
- What is a token? How AI reads text in chunksAsk a model how many r's are in "strawberry" and it often says two. The answer is three. It is not bad at counting; it never sees the letters. Text is chopped into tokens before the model reads it, and that single fact explains a surprising amount about how AI behaves, what it costs, and what it fails at.
- Why AI gives different answers: sampling and temperatureAsk a chatbot the same question twice and you often get two different answers. That is not a glitch. A language model does not choose the next word; it rolls weighted dice over thousands of options, and a setting called temperature controls how loaded those dice are. Here is how the last step of generation actually works.
- What is a large language model (LLM)? Complete guideLarge language models are the systems behind ChatGPT, Claude, and Gemini, the technology that made AI feel like it could talk, write, and reason. Underneath, an LLM does one deceptively simple thing: predict the next token. This guide explains what an LLM is, how it is built from pretraining through alignment, how it runs, what it can do, and where it falls short.
- What is natural language processing (NLP)?Every search, translation, voice assistant, and chatbot runs on natural language processing, the field of getting computers to work with human language. Its defining modern story is a quiet revolution: the dozens of separate, task-specific methods that made up NLP for decades collapsed into a single general approach, the large language model.
Further reading
- Sennrich, Haddow & Birch (2016), Neural Machine Translation of Rare Words with Subword Units — the paper that made byte-pair encoding standard in NLP.
- Gage (1994), A New Algorithm for Data Compression — BPE's origin, as a compression scheme, two decades before anyone applied it to language models.
- Kudo & Richardson (2018), SentencePiece — the language-independent tokenizer used by many non-GPT models.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming the ¾-of-a-word rule holds everywhere. It's an English-prose average. Japanese, Arabic, JSON, and code routinely cost two to three times more per visible character.
- Forgetting output tokens count too. Context limits and bills cover prompt plus answer, which is why long responses get truncated at what feels like an arbitrary point.
- Blaming the model for bad arithmetic when the tokenizer is the culprit. Numbers split in strange places, so digits the model needs to compare may not be in the same token at all.
At a glance
FieldLanguage & LLMs
Core ideasubword chunks of text
Rule of thumb~¾ of a word
Why it mattersthe unit of cost, context, and limits
DifficultyBeginner
Where this sits
2 concepts come first. Understanding it opens up 67 more.
2Levelsteps in
2Needs firstconcepts
67Opens up22% of 310
7Areascrosses fields
Learn these firstTokenization
Computed from the prerequisite graph, not assigned. How this works