Why AI aces the test and fails the variation

On a semantic parsing benchmark called COGS, transformer models scored between 96 and 99% on test items drawn from the same distribution as training.

On items built from exactly the same vocabulary and grammar, recombined in ways not seen during training, the same models scored 16 to 35%.

Nothing new appeared in the second set. Every word had been seen. Every construction had been seen. What changed was which words appeared in which constructions, and performance collapsed by sixty points.

A system that has learned a rule handles new combinations of familiar parts, because that is what having a rule means. A system that has learned which combinations occur handles the combinations that occurred. The gap between those two is called systematicity, it has been argued about since 1988, and it is the clearest available evidence about what these models are actually doing.

What compositionality is

The principle is old and it is easy to state.

The meaning of a complex expression is determined by the meanings of its parts and the way they are combined. This is usually attributed to Frege, and it explains something otherwise mysterious: how a finite vocabulary and a finite grammar let you understand sentences nobody has ever produced before.

Fodor and Pylyshyn drew the consequence in 1988. If your competence comes from grasping parts and rules, then it comes in systematic clusters. Anyone who understands "John loves Mary" understands "Mary loves John", not because they have encountered both, but because both are built from the same pieces by the same operation.

Their argument was that neural networks lack this by construction. A network that associates inputs with outputs has no guarantee that mastering one combination confers mastery of another, so it is not a plausible model of human cognition. That claim launched an argument still running thirty-eight years later.

What makes it a good argument, and better than the more famous philosophical objections to AI, is that it predicts something measurable.

What the benchmarks found

The prediction was tested, repeatedly, and the results were consistent for a long time.

SCAN, introduced in 2018, maps simple commands to action sequences: "jump twice" becomes JUMP JUMP. Trivial for a person who has learned "jump" and "twice" separately. Sequence models trained on some combinations and tested on others generalised poorly, succeeding mainly where test items closely resembled training items.

COGS, in 2020, moved to natural language semantic parsing and produced the numbers at the top of this article. The failure was not uniform noise. Models failed specifically on structural generalisation, including cases where a noun that had only appeared as a subject now appeared as an object, or where a verb underwent an alternation the model had seen for other verbs but not this one.

CFQ, on compositional question answering, found the same shape: strong performance when training and test compositions resembled each other, sharp decline when the structures were novel.

Across architectures, across tasks, across a decade: models trained on combinations handle combinations, and degrade on recombination.

The asymmetry that gives it away

One finding is worth more than the aggregate scores, because it discriminates between explanations.

Work on compositional instruction-following examined transfer in both directions. Training on higher-order compositions improved performance on lower-order ones. Training on lower-order compositions did not transfer upward.

A systematic learner should show both. If you have the rule, you have it for two elements and for four, and mastering the simple case gives you the machinery for the complex one. Human learners work this way: a child who can "skip" can be told to "skip backwards around a cone twice".

One-directional transfer is what a system looks like when it is learning patterns of increasing specificity rather than a rule. Complex examples contain simple ones as sub-parts, so training on the complex teaches the simple by inclusion. The reverse requires the thing under dispute.

This is the sharpest empirical result in the debate, and it is not widely quoted.

The finding that changes the picture

Then the picture complicated, in a way that should update anyone holding either position confidently.

Researchers found that prompting a model to decompose a problem before solving it produced near state-of-the-art out-of-distribution performance on exactly the benchmarks that had defeated earlier systems. First break the input into parts, then handle the parts in sequence, then combine. On CFQ and COGS, this closed much of the gap.

The conclusion the authors drew is the interesting one: systematicity is not reliably expressed by default, and it can be elicited.

That is a different claim from either side of the original argument. Fodor and Pylyshyn said the capacity is absent. The optimistic reply said it is present and the benchmarks are unfair. This says the capacity is there and is not the default behaviour, which means the model has compositional machinery it does not reliably deploy.

A second result pushes further. A meta-learning approach that trains a network on a dynamic stream of compositional tasks, rather than on a fixed dataset, achieved human-like systematicity in head-to-head behavioural comparison with people. The framing in that work is precise: symbolic models are perfectly systematic and rigid, standard networks are perfectly flexible and unsystematic, and the meta-learned system achieved both.

Taken together these say that systematicity is a property of how a system is trained and prompted rather than a property of the architecture, which is a substantial revision to a thirty-eight-year-old claim.

Why the encouraging results need scrutiny

Before concluding that the problem is solved, two confounds affect most of the recent optimistic findings, and they are not minor.

The novel combinations are frequently supplied in context. Evaluating a large model by showing it examples of the new combination and then testing it leaves very little generalising to do. The model is being asked to apply a pattern present in its prompt, which is a different and much easier task than deriving the combination from rules learned earlier.

The benchmarks use English words. A model that has pretrained on a large corpus already holds rich representations of the syntactic roles and semantic behaviour of the vocabulary being tested. When a benchmark asks whether the model can put a familiar noun into an unfamiliar syntactic position, the model may have seen exactly that during pretraining, in text nobody catalogued. The held-out set is only held out from the fine-tuning data.

That second point is the same difficulty that blocks progress elsewhere in this territory: testing whether a system generalises beyond its training requires knowing what its training contained, and for a multi-trillion-token corpus nobody does. Controlled experiments on small models trained on known data avoid this and raise the question of whether the results transfer.

The honest position is that recent results are encouraging and are measured through an instrument with known contamination, and that the size of the remaining effect is not established.

What this means if you are building something

The practical translation is more direct than the philosophy suggests.

Expect performance to fall on structural novelty, not just topical novelty. Teams test out-of-distribution by trying new subject matter. The failure mode here is different: same subject, same vocabulary, unfamiliar arrangement. A system handling "cancel the order and refund the customer" may fail on "refund the customer and cancel the order" if the second ordering was rare in training.

Decomposition prompting is not a trick, it is compensation for a known weakness. Asking a model to break a problem into parts before solving it works because it substitutes an explicit process for compositional machinery the model has and does not reliably use. This is why it helps most on exactly the tasks where compositional structure matters.

Benchmark scores tell you about the composition distribution they were drawn from. A model at 96% on your evaluation set may be at 30% on the same task with the pieces rearranged, and nothing in the score signals this. Constructing a held-out set that varies structure rather than content is the test that finds it, and almost nobody builds one.

And the asymmetry has an operational reading. If you are producing training or few-shot examples, complex ones teach the simple cases and simple ones do not teach the complex. Weight the examples accordingly.

A structural test set you can build this afternoon

The reason nobody catches this is that building the right held-out set requires a different move from the one teams habitually make, and the move is not obvious until it is described.

Take twenty inputs your system currently handles. For each, produce a variant that keeps every component and changes only how they are arranged.

Swap the arguments. If the working input is "move the invoice from draft to approved", try "move the invoice from approved to draft". Same entities, same relation, reversed roles. This is the direct analogue of the John-loves-Mary case.

Reverse the operation order. "Cancel the order and refund the customer" becomes "refund the customer and cancel the order". If the second ordering was rare in training, performance may fall even though both orderings are individually familiar.

Move an entity into a role it has not occupied. If a particular field only ever appears as something being read, make it the thing being written. Structural generalisation failures concentrate here.

Nest one level deeper. If the system handles "find the customer with the largest order", try "find the customer with the largest order placed before the policy change". Every element is familiar; the depth is not.

Substitute across a category boundary. Replace a noun with another noun of the same type that appeared in different constructions during training. This tests whether the role was learned or the pairing was.

Score the variants against the originals. A system that is compositional shows little gap. A system that has learned combinations shows a large one, and the size of the gap is the number worth knowing before deployment.

Two hours of work, and it surfaces a failure mode that ordinary out-of-distribution testing is structurally unable to find, because ordinary out-of-distribution testing varies content while holding structure fixed.

What is unresolved

Whether elicited systematicity is the same thing as having it. A system that behaves compositionally when prompted to decompose, and does not otherwise, has some capacity. Whether that counts as systematicity in the sense Fodor and Pylyshyn meant, or is a scaffold doing the compositional work externally, is a question about what the claim was ever about.

Whether the confounds can be removed at scale. The clean experiments run on small models with known training data. The interesting models have unknown training data. Nobody has a method that is both rigorous and applicable to frontier systems, and this may be a permanent condition rather than a temporary gap.

How systematic humans actually are. A long-running counter to Fodor and Pylyshyn holds that human compositional ability is less rule-like and more ragged than the argument assumes. If people also fail on some recombinations, the standard being applied to models is wrong, and the meta-learning result showing human-like rather than perfect systematicity supports this reading.

Whether it matters for capability. A system could be unsystematic and still handle every combination that occurs in practice, if the training distribution covers the space densely enough. Whether real task distributions are dense in that way, or whether the tail of unseen combinations is where the value is, is an empirical question that varies by domain.

The counter-argument

The benchmarks may test something narrow. SCAN in particular uses a synthetic grammar covering a small subset of English, and the original authors noted it was unclear whether progress on it would generalise to natural language. Building an argument about cognition on performance against a toy navigation language has obvious limits.

Large models were not the target. Fodor and Pylyshyn were arguing about connectionist architectures of the 1980s, and much of the damning benchmark evidence comes from sequence-to-sequence models that predate the current generation. Applying conclusions across that gap requires an argument nobody has fully made.

Failure on structural generalisation may be a training-data artefact. If the relevant combinations are rare in training because they are rare in language, a model reflecting that distribution is behaving correctly. The claim that it should generalise anyway assumes rules are the right description of the target, which is what the argument was about in the first place.

And elicitation may be enough. If decomposition prompting reliably produces compositional behaviour, the practical problem is solved regardless of what the model is doing internally. The philosophical question of whether it "really" has the capacity does not affect anyone building with it.

The short version

Transformer models scored 96 to 99% on a semantic parsing benchmark and 16 to 35% on the same task with familiar pieces recombined in unfamiliar ways. Nothing new appeared in the harder set: same vocabulary, same grammar, different arrangement. That gap is the systematicity question, first posed by Fodor and Pylyshyn in 1988, who argued that neural networks cannot generalise the way a rule-following system does and are therefore poor models of cognition.

Their argument is better than the more famous philosophical objections because it predicts something measurable, and for a decade the measurements agreed with them. SCAN found sequence models succeeding mainly where test items resembled training items. COGS found failure concentrated on structural generalisation. CFQ found sharp decline on structurally novel compositions. Across architectures and tasks, models handled combinations they had seen and degraded on recombination.

The sharpest single finding is an asymmetry: training on complex compositions improves simple ones, while training on simple ones does not transfer upward. A systematic learner should transfer both ways, since having the rule means having it at every arity. One-directional transfer is the signature of learning patterns of increasing specificity rather than a rule.

Then two results complicated it. Prompting models to decompose before solving produced near state-of-the-art out-of-distribution performance on the benchmarks that had defeated earlier systems, suggesting systematicity is not expressed by default and can be elicited. And a meta-learning approach achieved human-like systematicity in direct behavioural comparison. Together these suggest systematicity is a property of training and prompting rather than of architecture.

Two confounds should temper this. The novel combinations are often supplied in context, which leaves little generalising to do, and the benchmarks use English words the model has already seen in every syntactic position during pretraining, so the held-out set is only held out from fine-tuning. Testing whether a system generalises beyond its training requires knowing what its training contained, and at frontier scale nobody does.

Common questions

What is compositional generalization? The ability to understand or produce novel combinations of familiar components, following from the principle that the meaning of a complex expression is determined by its parts and how they combine. A system with compositional generalization handles arrangements it has never seen, because it has learned the parts and the rules rather than the arrangements. It is measured by training on some combinations and testing on structurally different ones built from the same elements.

Why do models score high on a benchmark and fail on variations? Because in-distribution and out-of-distribution performance measure different things. On the COGS benchmark, transformers scored 96 to 99% on test items drawn from the training distribution and 16 to 35% on items using the same vocabulary and grammar in unfamiliar structural arrangements. The score tells you about the composition distribution the evaluation was drawn from, and nothing in it signals how the model behaves when the pieces are rearranged.

What is systematicity in AI? The property that competence comes in clusters because it derives from parts and rules. Fodor and Pylyshyn's 1988 example: anyone who understands "John loves Mary" understands "Mary loves John", since both are built from the same primitives by the same operation. They argued neural networks lack this by construction, since associating inputs with outputs gives no guarantee that mastering one combination confers mastery of another.

Have neural networks solved the systematicity problem? Partly, and the qualification matters. Prompting models to decompose a problem before solving it produces near state-of-the-art out-of-distribution performance on benchmarks that defeated earlier systems, which suggests the capacity exists and is not the default behaviour. A meta-learning approach trained on a dynamic stream of compositional tasks achieved human-like systematicity in behavioural comparison with people. Both results suggest systematicity depends on training and prompting rather than on architecture.

Why should I distrust compositional generalization results on large models? Two confounds. The novel combinations are frequently shown in context, so the model applies a pattern present in its prompt rather than deriving it from earlier learning, which is a much easier task. And the benchmarks use English words whose syntactic roles the model already learned during pretraining, so the held-out set is only held out from fine-tuning. Establishing that a combination is absent from a multi-trillion-token corpus is not currently possible.

What is the asymmetry finding? That training on higher-order compositions improves performance on lower-order ones, while training on lower-order compositions does not transfer upward. This matters because a systematic learner should transfer in both directions: having the rule means having it at any complexity. One-directional transfer is what you see when a system learns patterns of increasing specificity, since complex examples contain simple ones as sub-parts while the reverse requires the capacity under dispute.

How do I test my system for this? Build a held-out set that varies structure rather than content. Teams usually test out-of-distribution by introducing new subject matter, which measures topical novelty. The failure here is structural: same domain, same vocabulary, unfamiliar arrangement. Take working inputs and rearrange the order of operations, move an entity into a role it has not occupied, or nest a construction one level deeper, keeping every component familiar.

Does decomposition prompting actually help? Yes, and it is worth understanding why rather than treating it as a trick. Asking a model to break a problem into parts and solve them in sequence substitutes an explicit process for compositional machinery the model possesses and does not reliably deploy. That is why it helps most on tasks where compositional structure is the difficulty, and why the improvement is largest exactly on the benchmarks designed to test it.