Why AI is bad at math, and which failures are permanent

Ask a capable model whether 9.11 is greater than 9.9 and there is a real chance it says yes.

Ask the same model to identify the first digit of a large product and it will often be right. Ask for the last digit, which is arithmetically the easier question, and it will often be wrong.

That inversion is the tell. A system reasoning about numbers finds the easy question easy. A system that has learned what number-shaped text looks like finds whichever question matches its training distribution easy, and the difficulty ordering it exhibits has no relationship to the difficulty ordering of the mathematics.

Some AI failures shrink with every model release. Others are properties of how text is represented before the model sees it, and those do not improve with scale. Telling them apart determines whether you should wait, prompt differently, or stop asking the model to do it at all.

The two kinds of failure

The distinction is worth stating precisely because almost all practical advice depends on which one you are facing.

Capability failures are cases where the model lacks the knowledge, reasoning depth or context to get it right. These respond to better models, more context, better prompting, and more compute at inference time. They have been shrinking steadily.

Representational failures are cases where the information required to answer correctly was destroyed before the model received it. No amount of reasoning recovers information that is not in the input. These do not improve with scale, and the evidence on this is now direct rather than inferred.

The distinction matters because the two produce identical symptoms. A wrong answer looks like a wrong answer. But one of them is a reason to try a stronger model and the other is a reason to stop trying.

What the model actually receives

A number does not arrive at the model as a quantity. It arrives as tokens, chunked by a vocabulary built for the frequency statistics of language rather than the structure of arithmetic.

The consequences are more severe than the usual explanation suggests.

The same number splits differently in different contexts. A figure like 87439 may become one grouping in one sentence and a different grouping in another, depending on surrounding text. Positional value is not consistently perceptible, because the boundaries that define position are not consistent.

Magnitude is not encoded. Splitting 100400 into two chunks carries nothing indicating that the first represents hundreds of thousands. The model has to infer magnitude relationships statistically, from examples, rather than reading them off the representation.

Equivalent notations are unrelated. 12,345 and 12345 and 12 345 are the same quantity and three different token sequences with three different sets of next-token probabilities. The same holds for 0.007, 7.0e-3 and 7×10⁻³. Nothing in the representation says these are the same number.

Surface changes alter segmentation. A typo, a different space character, or an unusual Unicode variant produces a different token sequence and therefore different behaviour, on input a human would read identically.

That last one generalises beyond arithmetic and is worth holding: the model's input is not the text you see, and two strings that look the same to you may be entirely different objects to it, which is also why some languages cost more.

The experiment that settles it

For a long time the tokenization explanation was plausible and circumstantial. Then someone tested it directly.

Researchers manipulated how numbers were grouped into tokens while holding everything else constant, and measured what happened. Accuracy shifted substantially and the distribution of errors changed shape. The model's arithmetic performance depended on the segmentation of the input.

The conclusion the authors drew is the important one: model reasoning on these tasks follows tokenizer-imposed structure rather than learned abstract computational principles. If the model had internalised addition, the grouping of the input would be irrelevant. It is not irrelevant, so it has not.

And the finding that decides the practical question: these failures persist as parameter counts increase. Scaling does not systematically resolve reasoning deficiencies rooted in representational mismatch, because the mismatch happens before the parameters are involved.

The inverted difficulty ordering

The single sharpest diagnostic in this literature, and the one to reach for when classifying an unfamiliar failure.

Models have been found to fail at determining the last digit of a product while succeeding at identifying the first. Arithmetically this is backwards: the last digit depends only on the last digits of the operands and is trivially computable, while the first digit depends on the full magnitude of the result.

A system executing an algorithm finds the algorithmically easy case easy. A system matching patterns finds the case that resembles its training data easy, and the two orderings coincide only by accident.

This gives you a test. Take a task where you can vary difficulty along a dimension the mathematics cares about, and see whether performance tracks it. If it does, you are looking at a capability limit that may improve. If performance is uncorrelated with real difficulty, or inverted, the model is not doing the thing you think it is doing and no amount of scale will make it start.

What is representational, and what is not

Sorting the common failures.

Representational, and unlikely to improve with scale:

Counting characters. The model does not see characters. Counting the letters in a word requires information the tokenizer discarded. This remains unreliable even in models trained explicitly to reason.

Reversing or manipulating strings at character level. Same cause. The operations are defined over units the model has no direct access to.

Multi-digit arithmetic carry propagation. Carry depends on digit position, and position is not consistently represented. Errors here characteristically produce a result of the right length and plausible appearance, which is the worst kind of failure because it survives casual inspection.

Decimal magnitude comparison. The 9.11 versus 9.9 case. The comparison requires reading the fractional part positionally, and the tokenisation may not preserve which digit occupies which place.

Anything sensitive to exact formatting. Since notation determines segmentation, tasks where equivalent formats must be treated identically are fighting the representation.

Capability-limited, and improving:

Multi-step word problems. These require reasoning about a situation, and reasoning has improved substantially. The arithmetic inside them has not, which is why these often fail at the final calculation after correct setup.

Applying a stated method. Give the procedure explicitly and models follow it better than they used to.

Recognising which operation applies. Improving, and largely a comprehension task rather than a computational one.

The practical reading: failures in the first list are permanent absent architectural change. Failures in the second are worth revisiting after a model upgrade.

What actually helps

Ordered by how much they return, and the first is not a prompting technique.

Do not ask the model to compute. Give it a calculator. Tool use resolves the representational problem entirely by moving the computation to a system that represents numbers as numbers. The literature notes, somewhat ruefully, that research into intrinsic arithmetic capability is now overshadowed by reliance on external tools, and for anyone building something this is the right answer rather than a disappointment.

Force step-by-step decomposition for anything multi-digit. Where the model must compute, making it write out intermediate steps converts one hard token prediction into several easier ones. This helps measurably and does not fix the underlying representation.

Control the formatting. Present numbers consistently, in one notation, with consistent separators. Since format determines segmentation, inconsistent formatting introduces variance for no reason.

Consider digit-level presentation. Research on arithmetic-specific training has found character-level tokenization outperforming subword for these tasks, and inserting explicit position markers helping further. One model fine-tuned on arithmetic outperformed a much larger general model on large-number addition, attributed substantially to its tokenizer treating digits consistently. You cannot change a model's tokenizer, and you can sometimes change how you present numbers to it.

Verify rather than trust. Arithmetic errors produce plausible-looking output. A result of the correct length and magnitude that is wrong in the middle digits will pass any review that is not an actual recomputation.

Five failures, sorted

The frame is only useful if it produces different actions, so here are five real complaints classified, with the action each implies.

"It gets long multiplication wrong." Representational. Carry propagation needs positional information the tokenisation does not reliably preserve, and errors characteristically produce a result of correct length that is wrong in the middle. Action: give it a calculator. Do not wait for a better model, and do not prompt harder.

"It miscounts items in a list." Mixed, and the split is diagnostic. Counting discrete items in structured text is closer to a capability limit and has improved. Counting characters within words is representational and has not. Action: if the units are tokens or larger, retry with a better model. If they are characters, stop.

"It fails when I paste data from a spreadsheet." Representational, and usually fixable at your end. Copied data carries locale separators, non-breaking spaces and inconsistent decimal marks, all of which change segmentation. Action: normalise the formatting before it reaches the model. This is the cheapest fix on the list and almost nobody does it.

"It sets up the word problem correctly and then gets the answer wrong." Both, in sequence, which is why it is confusing. The setup is a comprehension task that has improved; the calculation is representational and has not. Action: let the model do the setup and hand the arithmetic to a tool. The failure is exactly at the boundary between the two categories.

"It contradicts itself about the same number in one response." Representational. The same quantity written two ways is two unrelated token sequences, so consistency across them was never guaranteed. Action: pin the notation in the prompt and ask for one canonical form.

The pattern across all five: the classification tells you whether to change the model, change the input, or change the architecture. Those are three different budgets and teams routinely spend the first when the answer was the second.

The general lesson beyond arithmetic

The frame here extends past numbers, and this is the part worth taking away.

Every task has a representation, and some tasks require information that the representation does not carry. When that happens, the failure is not a reasoning failure and does not respond to reasoning improvements.

The diagnostic question is: could a perfect reasoner solve this from what the model actually receives? If a person given only the token sequence, with no access to the underlying characters, could not count the letters in a word, then neither can the model, and its failure to do so tells you nothing about its reasoning.

This reframes a whole class of complaint. Much of the disappointment directed at these systems is disappointment that they cannot do things the input does not permit, and separating those cases from real capability gaps makes both easier to reason about.

What is unresolved

Whether architectural change fixes it. Byte-level and character-level models remove the representational problem at the cost of much longer sequences, and specialised numerical encodings have been proposed. Whether any of these becomes viable at frontier scale is open, and current frontier systems all still tokenize.

How numbers are represented internally. Probing work suggests models encode numerical values in ways partially recoverable by linear probes, though not accurately enough to explain how exact operations sometimes succeed. The mechanism by which a model gets arithmetic right when it does is not well understood, which limits how confidently anyone can predict when it will fail.

Whether tool use is a solution or an evasion. Delegating computation works and it means intrinsic numerical capability stops improving because nobody needs it to. If some downstream capability depends on numerical competence in ways tool use does not cover, that would surface later and unpredictably.

How much of reasoning-model improvement is real here. Models trained to reason at length do better on numerical tasks, and it is not established whether that reflects better computation or more opportunities to pattern-match a correct-looking answer. The counting failures persisting in reasoning models suggest the latter contributes.

The counter-argument

Models are much better at this than they were. Arithmetic performance has improved substantially, large models handle many operations reliably, and an article emphasising failure can leave a misleading impression. For everyday quantities, current systems are usually right.

Humans are also bad at arithmetic. People make carry errors, misread decimals and count characters wrongly. Holding models to a standard of exact computation, when the comparison class is a person who would also reach for a calculator, may be the wrong benchmark.

Tokenization may not be the whole story. The literature identifies positional encoding and training data composition alongside it, and attributing everything to tokenization is a simplification. The digit-grouping experiments establish that tokenization matters and not that it is the only thing that does.

And representational limits are not obviously permanent. Calling a failure representational assumes the representation is fixed. Tokenization is a design choice with active alternatives, and a shift in that choice would move several items from the permanent list to the improving one.

The short version

Models can identify the first digit of a product while failing on the last, which is arithmetically the easier question. That inversion indicates the difficulty ordering follows training distribution rather than mathematical structure, which is the signature of pattern matching rather than computation.

Two kinds of failure produce identical symptoms and need opposite responses. Capability failures come from insufficient knowledge or reasoning depth and shrink with better models. Representational failures come from information destroyed before the model received it, and no reasoning recovers what is not in the input.

Numbers reach the model as tokens chunked for language frequency rather than arithmetic structure. The same number splits differently in different contexts, so positional value is not consistently perceptible. Magnitude is not encoded, since splitting a figure into chunks carries nothing about place value. Equivalent notations produce unrelated token sequences. And surface changes such as typos or unusual space characters alter segmentation, so two strings that look identical to you may be different objects to the model.

The direct evidence is that manipulating digit grouping while holding everything else constant produces substantial accuracy shifts and different error distributions, meaning the arithmetic follows tokenizer-imposed structure rather than an internalised method. These failures persist as parameter counts increase, because the mismatch occurs before the parameters are involved.

Representational and unlikely to improve: character counting, string reversal, multi-digit carry propagation, decimal magnitude comparison, and anything sensitive to exact formatting. Capability-limited and improving: multi-step word problems, applying a stated method, recognising which operation applies.

The diagnostic that generalises past arithmetic: could a perfect reasoner solve this from what the model actually receives? If not, the failure is not a reasoning failure and will not respond to reasoning improvements. And the first-line fix is not a prompt. It is a calculator.

Common questions

Why is AI bad at math? Largely because numbers reach the model as tokens chunked for language frequency rather than arithmetic structure, so positional value and magnitude are not reliably represented. The same figure can split differently in different contexts, equivalent notations produce unrelated token sequences, and carry propagation depends on positional information the representation does not consistently preserve. This is a representational limit rather than a reasoning limit, which is why it does not improve much with model size.

Why does AI say 9.11 is greater than 9.9? Because comparing decimals requires reading the fractional part positionally, and tokenization may not preserve which digit occupies which place. The model is matching patterns over token sequences rather than comparing quantities, and "11" appearing larger than "9" is a pattern that exists in the text distribution. It is the same underlying cause as multi-digit arithmetic errors.

Will better models fix AI arithmetic? Partly and not entirely. Failures rooted in representation persist as parameter counts increase, because the information loss happens in tokenization before the model processes anything. Direct experiments manipulating digit grouping show accuracy shifting substantially, which indicates the arithmetic follows tokenizer structure rather than a learned method. Reasoning improvements help with multi-step word problems while leaving the computation inside them unreliable.

Why can't AI count letters in a word? Because it does not see letters. Text is split into subword tokens before the model receives it, so individual characters are not available unless a character happens to be its own token. Counting characters requires information the tokenizer discarded, and no amount of reasoning recovers information that is not in the input. This remains unreliable even in models trained specifically to reason at length.

How do I tell if an AI failure will improve with a better model? Ask whether a perfect reasoner could solve the task from what the model actually receives. If the required information was destroyed in tokenization, the answer is no and scale will not help. A second test: vary difficulty along a dimension the task cares about and see whether performance tracks it. If performance is uncorrelated with real difficulty, or inverted, the model is pattern matching rather than computing.

What actually fixes AI arithmetic errors? Tool use, first and by a wide margin. Giving the model a calculator moves computation to a system that represents numbers as numbers and resolves the representational problem entirely. Where the model must compute, forcing step-by-step decomposition converts one hard prediction into several easier ones. Beyond that: present numbers in consistent notation since format determines segmentation, and verify results rather than trusting them, because arithmetic errors produce output of plausible length and magnitude.

Does formatting numbers differently change AI accuracy? Yes, measurably. Values like 12,345 and 12345 and 12 345 are the same quantity and three different token sequences with different next-token probabilities. The same applies to 0.007 against 7.0e-3. Small surface changes including typos, alternative space characters and unusual Unicode variants trigger different segmentations and produce different behaviour on input a human reads identically.

Is tokenization the only cause of these failures? No, and attributing everything to it oversimplifies. The literature identifies positional encoding and training data composition as contributing causes alongside tokenization. The digit-grouping experiments establish that tokenization has a substantial causal role and do not establish that it is the sole one. Work on specialised numerical encodings and character-level tokenization suggests changing the representation helps considerably without resolving arithmetic entirely.