What is a token? How AI reads text in chunks

Ask a large language model how many times the letter "r" appears in "strawberry" and there is a good chance it will confidently answer two. The correct answer is three. This looks like a stupid mistake from a system that can write working code and explain quantum mechanics, and it puzzles people. The explanation is one of the most useful things you can know about how these models work, and it has nothing to do with counting. The model gets it wrong because it never sees the letters at all.

Before a language model reads your text, the text is chopped into chunks called tokens, and those chunks, not the individual letters, are what the model actually processes. Tokens are the atoms of an LLM's perception, the smallest units it perceives text in, and almost everything about how a model behaves, what it costs, how much it can read at once, and the specific things it inexplicably fails at, traces back to them. This piece explains what a token really is, why models use these odd in-between units instead of words or letters, how the chunks get made, and why understanding tokens quietly makes you better at using AI and sharper about how it works.

What a token actually is

Start with the mechanism, because it is simple and most people never see it. A computer cannot process text directly; it works with numbers. So the first thing that happens to your prompt, before any intelligence is involved, is tokenization: the text is split into tokens, and each token is mapped to an integer id from a fixed list called the vocabulary. The word "Hello" might become token number 9906; a space followed by "there" might be a single token with its own id. Your sentence becomes a sequence of integers, and it is those integers, indexing into a big table of embeddings, that the model actually reads. The model never sees your characters. It sees a list of id numbers. When it replies, it generates ids, and a reverse step turns them back into text for you.

The important and slightly surprising part is what a token is. It is not a word, and it is not a letter. It is usually a subword, a fragment that might be a whole common word, a piece of a longer one, a space-plus-word, or occasionally a single character. "Hello there" splits into a few tokens; "strawberry" splits into something like "straw" and "berry"; a long rare word like "antidisestablishmentarianism" shatters into seven or eight fragments. A rough rule for English is that one token is about four characters, or roughly three-quarters of a word, so a thousand tokens is around 750 words. But it is only a rule of thumb, and the exact split depends on the model.

Why subwords, and not words or letters

The obvious question is why models use these awkward fragments instead of something intuitive like whole words or individual letters. Both of the intuitive options were tried, and both fail in instructive ways.

Using whole words as tokens seems natural, but it breaks down fast. The vocabulary would need to be enormous to cover every word, and it still could not handle words it had never seen: a new slang term, a typo, a rare name, a chemical compound, or any word in a language it was not built for would have no token at all. A word-level model is brittle at the edges of language, which is most of the interesting part.

Using individual characters solves that (any text is just letters, so nothing is unrepresentable), but it creates the opposite problem. Sequences become enormously long, since every letter is a separate token, which wastes the model's limited capacity to look at text and makes it harder to see patterns across a passage. Character-level models spend all their effort on spelling and have little room left for meaning.

Subword tokenization is the middle ground that won, and it is clever for a reason worth appreciating. Common words get their own single token, so frequent text stays compact. Rare or unseen words get broken into familiar smaller pieces, so the model can always represent anything by combining known fragments, even a word it has never encountered. The vocabulary stays a manageable size (typically tens of thousands to a few hundred thousand tokens), and sequences stay reasonably short. You get the coverage of characters and most of the compactness of words at once.

How the chunks get made: byte pair encoding

The most common way to decide on those subword units is an algorithm called byte pair encoding, or BPE, and its logic is easy to follow. It began, oddly, as a data-compression technique, and it was adapted to language.

BPE builds a vocabulary from the bottom up. Start with every individual character as a token. Then look through a large amount of training text and find the pair of tokens that appears together most often. Merge that pair into a single new token. For example, if "t" and "h" occur together constantly, merge them into "th". Now find the next most frequent pair, which might be "th" plus "e", and merge that into "the". Repeat this thousands of times, each merge adding one token to the vocabulary, until you reach the target size. The result is a vocabulary where the most common letter sequences have become single tokens while rare combinations remain splittable into pieces. Frequent words end up as one token; unusual ones stay as several. The whole scheme is learned from the statistics of real text, which is why a tokenizer trained mostly on English carves English efficiently and stumbles on other languages.

Back to strawberry: why tokens explain the failures

Now the strawberry mystery dissolves, and it reveals something deeper. When the model receives "strawberry", it does not see s-t-r-a-w-b-e-r-r-y. It sees two opaque tokens, roughly "straw" and "berry", each just an id number pointing to a learned meaning. The letters inside those tokens are invisible to it. Asking it to count the r's is like asking you to count the r's in two flashcards showing pictures rather than spellings. The information is simply not there in the form the question needs.

This is the clean demonstration of a broad truth: an LLM's alphabet is its token vocabulary, and that alphabet does not decompose into letters. Any task that requires operating on individual characters is uphill for the architecture, not because the model is dim but because it perceives text in chunks that hide the characters. Counting letters, spotting anagrams, judging whether two words rhyme, reversing a string, fixing individual typos, working out a spelling puzzle: these are the tasks models are strangely bad at, and they are all character-level tasks fighting against a token-level perception. Once you see this, a whole category of "why is the AI so dumb about this simple thing" stops being mysterious. It is the same root cause every time. (Models handle these better now than they used to, partly by learning workarounds and partly with tool use, but the underlying tension never fully goes away.)

Why tokens are also your bill and your limits

Tokens are not just a technical curiosity; they are the unit of two things you deal with constantly: cost and capacity.

Every model has a context window, the maximum amount of text it can consider at once, and that maximum is measured in tokens, counting both your input and the model's output together. When you read that a model has a context window of, say, 200,000 or a million tokens, that is how much text (in these subword units) it can hold in view at one time. Exceed it and something has to be dropped or the text has to be split, which is exactly why RAG systems break documents into chunks sized to fit. Tokens are the currency of the context window.

They are also the currency of the bill. Model APIs charge per token, usually with separate rates for input and output, so the length of both your prompt and the response directly sets what you pay. This is why the same task can cost very different amounts depending on how verbose the prompt is, and why estimating token counts matters for anyone building on these models at scale. When you understand that a page of text is roughly 500 tokens, the economics of an AI product become something you can actually reason about.

The hidden unfairness: not all languages tokenize equally

One consequence of tokens deserves special mention, because it is a real fairness issue hiding in the plumbing. Because tokenizers are trained mostly on English-heavy data, they carve English efficiently but split many other languages into far more tokens for the same meaning. A sentence in Hindi, Arabic, or Chinese can take two to four times as many tokens as its English translation.

That multiplier has direct consequences. Speakers of those languages pay more for the same content (more tokens means a bigger bill), fit less of their text in the context window, and sometimes get lower-quality output because the model is working with more fragmented input. A property that looks like a dry technical detail turns out to distribute cost and quality unevenly across the world's languages, and it is one of the quieter equity problems in how these systems are built.

The short version

A token is the unit of text a language model actually processes: text is split into tokens (usually subword fragments), each mapped to an integer id, and the model reads those ids rather than your letters. Subwords won out over whole words (too many, can't handle the unseen) and individual characters (sequences too long), giving broad coverage and reasonable length at once, and the units are usually built by byte pair encoding, which merges the most frequent character pairs until it has a vocabulary. Because the model perceives text in these chunks and never sees the letters inside them, character-level tasks like counting the r's in "strawberry" are hard for it. And because tokens are the unit of both the context window and API billing, they quietly govern what AI can read and what it costs, unevenly across languages.

The idea to hold onto is that a model does not read text the way you do; it reads a sequence of tokens, chunks of text that are not words and not letters, and that single fact explains its costs, its limits, and the odd little tasks it fails at. The tokenizer is the silent boundary between your text and the model, and most people never look at it. Looking at it, even once, changes how you understand everything the model does next.

Common questions

What is a token in AI? A token is the basic unit of text that a language model processes. Before the model reads your text, the text is split into tokens, usually subword fragments, and each token is converted into an integer id from a fixed vocabulary. The model operates on those id numbers, not on your original letters. A token is not a word and not a character but something in between: a common word might be one token, while a rare word breaks into several. As a rough rule, one token is about four English characters or three-quarters of a word.

Why can't AI count the letters in a word like "strawberry"? Because the model never sees the individual letters. "Strawberry" is split into a couple of tokens (roughly "straw" and "berry") before the model reads it, and each token is just an id pointing to a learned meaning, with the letters inside hidden. Asking it to count the r's is like asking someone to count letters in a word they can only see as two opaque symbols. This is why models struggle with character-level tasks generally: counting letters, anagrams, rhymes, and spelling puzzles all fight against the fact that the model perceives text in chunks, not letters.

Why do LLMs use subword tokens instead of words or letters? Whole words fail because the vocabulary would be enormous and still could not handle words the model never saw, like typos, new slang, or rare names. Individual characters fail because sequences become extremely long, wasting the model's limited capacity and making patterns harder to see. Subword tokens are the middle ground: common words become single tokens for compactness, while rare words break into familiar smaller pieces so the model can represent anything by combining known fragments. This gives broad coverage and reasonable sequence length at the same time.

What is byte pair encoding (BPE)? Byte pair encoding is the most common algorithm for creating subword tokens. It starts with individual characters, then repeatedly finds the most frequently occurring pair of tokens in the training text and merges them into a single new token, continuing until it reaches a target vocabulary size. Common letter sequences (like "th", then "the") become single tokens, while rare combinations stay splittable. Originally a data-compression technique, BPE now underlies the tokenizers of most modern language models, which is why frequent words are one token and unusual ones are several.

How many tokens is a word? For English, roughly one token is about four characters or three-quarters of a word, so 1,000 tokens is around 750 words and a typical page is about 500 tokens. It is only a rule of thumb: common words are often a single token, while long or rare words split into several. The exact count depends on the model's tokenizer, and it differs significantly across languages, with many non-English languages using two to four times as many tokens for the same meaning.

How do tokens affect cost and context limits? Tokens are the unit of both. A model's context window, the maximum text it can consider at once, is measured in tokens and counts your input plus the model's output together, so exceeding it forces text to be dropped or split (which is why RAG chunks documents to fit). API pricing is also per token, usually with separate input and output rates, so the length of your prompt and the response directly determines what you pay. Understanding that a page is roughly 500 tokens makes both the limits and the economics of using AI much easier to reason about.

Why do non-English languages cost more tokens? Because tokenizers are trained mostly on English-heavy data, so they split English efficiently but break many other languages into more, smaller tokens for the same meaning. A sentence in Hindi, Arabic, or Chinese can take two to four times as many tokens as its English equivalent. This means speakers of those languages pay more per API call, fit less of their text in the context window, and can get lower-quality results from more fragmented input. It is a real fairness issue built into the tokenization layer rather than the model's intelligence.