The model was right. Acting on it would have killed people.

In the 1990s a team building models to predict pneumonia mortality found something odd in the output. Patients with a history of asthma had lower risk of dying than patients without it.

This is backwards. Asthma damages the lungs. A pneumonia patient with asthma should be in more danger, not less. But the model was not confused and the data was not wrong. In the hospitals that produced the training data, patients who arrived with pneumonia and a history of asthma were recognised as high risk and sent directly to intensive care, where they received aggressive treatment. The aggressive treatment worked. Their observed mortality was lower.

The model learned the effect of the treatment and reported it as a property of the patient. Had it been deployed to triage, it would have recommended sending asthmatic pneumonia patients home, removing the very care that produced the number it had learned from.

Every standard machine learning model answers one question: what tends to occur together. Almost every decision anyone makes with a model asks a different question: what happens if I act. The two questions have different answers, nothing in the usual toolkit flags when you have swapped one for the other, and the failure is invisible in every accuracy metric you have.

Why this is not a bug

It is tempting to file the asthma case as a data-quality problem, or as something a better model would have caught. It is neither, and understanding why is the whole subject.

The model performed its job correctly. It was asked to find patterns that predict mortality, and it found one. The pattern is real, reproducible, and was later confirmed with different modelling approaches. Any model trained on that data would find it, because it is in the data.

The problem is that the data records a world in which a policy was already operating. Doctors were already treating asthmatic pneumonia patients aggressively. The observed outcomes are outcomes-under-that-policy, and a model trained on them learns the policy's effects along with everything else, without any marker distinguishing the two.

This generalises well beyond medicine. Any dataset drawn from an operating business records the effects of that business's existing decisions. Customers who received a retention offer churn less, because the offer worked. Loan applicants who were approved default rarely, because approval was selective. Ads shown to interested users convert well, because targeting put them there. In each case the correlation is genuine and using it to decide who gets the offer, the loan, or the ad will produce a worse outcome than the model predicts.

The technical name is confounding: a third factor influences both the thing you measured and the outcome. In the pneumonia case the confounder is treatment intensity. In the churn case it is whoever decided which customers to target.

Pearl's ladder, which is the standard framing

Judea Pearl, who received the Turing Award in 2011 for this work, organised the distinction into three levels. The framing is worth knowing because it makes clear that these are not degrees of rigour but different questions requiring different tools.

Rung one, association. What tends to occur together. Patients on this drug have better outcomes. Every standard machine learning model lives here, and it is where prediction happens. Adequate for spam filters, image classifiers, demand forecasts, and anything where you observe rather than intervene.

Rung two, intervention. What happens if I act. If we give this drug to this patient, does the outcome improve. This requires knowing what changes when you change something, which observation alone does not supply. Pearl writes it with the do-operator, distinguishing the probability of an outcome given that you observe a treatment from the probability given that you administer it. Those are different quantities and confusing them is the error described above.

Rung three, counterfactual. What would have happened otherwise. Would this patient have recovered without the drug. This is the hardest rung because the alternative was never observed, and it is what a great deal of practical reasoning about responsibility, credit and blame actually requires.

The distance between rung one and rung two is where decisions go wrong at scale. Models sit on rung one. Decisions live on rung two. The gap is not bridged by more data, better architecture, or higher accuracy, because those improve your answer to a question you were not asking.

The most common mistake is the one that feels careful

Ask a data scientist how to handle confounding and most will say: control for it. Include the confounding variables in the model, and their influence is accounted for.

This is correct sometimes and it is wrong in a way that catches almost everyone, because the intuition it rests on generalises badly. The intuition is that adjusting for more variables is safer, in the way that a more thorough audit is safer. It is not. Which variables to adjust for is a structural question with a right answer, and including everything available reliably produces bias rather than removing it.

The mechanism is a collider. If two variables both influence a third, then conditioning on that third variable creates an association between them that does not exist otherwise. Suppose talent and luck are independent in the population, and either one can get you hired. Among people who were hired, talent and luck will be negatively correlated, because someone who got in without much luck probably had talent, and vice versa. Nothing caused that relationship except the act of looking only at hired people.

This is not exotic. Selection into your dataset is a collider whenever the selection depends on both your treatment and your outcome, which is common. Studies of hospitalised patients condition on hospitalisation. Studies of active users condition on retention. Studies of successful projects condition on success. Each introduces relationships that will look like findings.

The consequence is that adjustment is a decision requiring a hypothesis about the causal structure. There is no procedure that reads the right adjustment set off the data, because the data is compatible with many structures. You have to draw the graph, and drawing the graph requires knowing something about the domain that is not in the file.

The gap between what people know and what they do

Everyone in this field can recite that correlation does not imply causation. It is one of the few statistical facts that has escaped into general circulation. The recitation has not changed practice.

A systematic review of ninety machine learning studies on immune checkpoint inhibitors found that not one incorporated causal inference. A parallel review of thirty-six retrospective studies modelling melanoma found the same: zero. These are studies whose entire purpose is informing treatment decisions, published by researchers who know the distinction, using methods that cannot answer the question being asked of them.

The paper naming this calls it a knowledge-practice gap, and the phrase is generous. The reasons are structural rather than intellectual.

Prediction is measurable and causation is not. You can compute accuracy on a held-out set and put a number in a paper. A causal estimate has no equivalent ground truth, because you never observe the counterfactual. Validation depends on domain expertise, refutation tests and sensitivity analysis, which are harder to run, harder to publish and harder to defend.

Assumptions must be stated and defended. A predictive model can be presented on its performance. A causal estimate requires you to write down what you assumed about the structure of the world and argue for it, knowing those assumptions cannot be verified from the data. That is professionally uncomfortable in a way that reporting an AUC is not.

The tooling is worse. Predictive machine learning has a decade of mature, well-documented libraries with sensible defaults. Causal tooling has improved considerably and remains further from the point where a competent practitioner can use it without understanding it, which is the standard the predictive stack now meets.

None of these makes the practice acceptable. They explain why the gap persists despite everyone knowing better.

A test you can run this week

The abstract version of this is easy to nod at and hard to act on, so here is a concrete diagnostic that takes an afternoon and finds the problem more often than people expect.

Take any model currently informing a decision at your organisation. Write down two sentences.

Sentence one: what the model predicts. "This customer has a 73% probability of churning in the next quarter."

Sentence two: what someone does because of it. "So we send them a discount."

Now ask whether the training data contains any examples of the action in sentence two being taken. If it does not, the model has no information about what that action does, and its output is being used to justify an intervention it knows nothing about. If it does contain such examples, ask who decided which customers received the action historically, because that decision is now baked into the correlation the model learned.

In most organisations this exercise finds at least one model where sentence two does not follow from sentence one at all. Nobody notices because the model's accuracy is monitored and the intervention's effect is not.

The follow-up question is the useful one: what would it cost to randomise this decision for two weeks? Frequently the answer is very little, and the resulting estimate is worth more than the model.

Why the tooling gap persists

One more structural reason, worth separating from the others because it is fixable and the others are not.

Predictive machine learning has an unusually clean interface. You supply examples, you receive a model, you evaluate it on held-out data, and the whole loop is automatable. That property is why the ecosystem industrialised: a library can present a sensible default because the problem shape is the same every time.

Causal inference has no such interface. The right method depends on what confounds what, which instrument is available, whether an arbitrary threshold exists in your process, and what you are willing to assume. None of that is in the data, so no library can choose it for you. A causal package can implement estimators and cannot supply the argument that makes an estimator appropriate, and the argument is the hard part.

The consequence is that causal work does not scale the way predictive work does, and it stays specialist. Some of that is irreducible. But a meaningful share is that the field has invested heavily in estimation and much less in the workflow around it: expressing assumptions in a form a colleague can review, checking a proposed adjustment set against a stated graph, running refutation tests automatically. Those are tractable engineering problems and they are where practical progress would come from.

Where this bites outside medicine

The clinical examples are vivid, which makes them easy to file as somebody else's problem. The same failure runs through ordinary commercial work, usually undetected because there is no coroner.

Churn and retention. A model identifies customers likely to leave. The retention team offers them a discount. Some of those customers were never going to leave and have now been trained to threaten departure. The model was accurate about who would churn and silent about who would respond to an offer, which is the only question that mattered.

Pricing. Historical data shows higher prices associated with higher revenue in some segment. That segment probably contained customers with less price sensitivity, which is why they were charged more. Raising prices for everyone will not reproduce the pattern.

Hiring. A model finds features predicting good performance among current employees. Those features predict performance among people who were hired and stayed, which is a collider twice over. Deploying it selects for people resembling those who survived a process, which is not the same as people who would perform.

Marketing attribution. The last channel touched before a purchase gets the credit. It is generally the channel most correlated with intent rather than the one that created it, and shifting budget toward it reliably underperforms the model's forecast.

The pattern is identical each time. A model is accurate. Someone acts on it. The result underperforms and nobody can say why, because the metric that would have shown the problem was never computed and there is no error to inspect.

What actually works

The methods are not new and most predate machine learning. That is not a weakness.

Randomise if you possibly can. Assigning the treatment at random breaks the link between treatment and everything else, which is what makes the comparison valid. This is what an A/B test is, and its whole value lies in the randomisation rather than in the statistics that follow. An experiment on a few thousand users answers a question that no amount of observational data with millions of rows can answer, and organisations routinely make the opposite trade because the observational data is already there.

When you cannot randomise, the observational toolkit exists. Difference-in-differences compares changes over time between exposed and unexposed groups, so anything stable about the groups cancels. Instrumental variables use something that affects treatment but not the outcome directly, which is rare and precious when found. Regression discontinuity exploits an arbitrary threshold, comparing those just above and just below, who differ only by the cutoff. Propensity methods construct comparable groups by modelling who received treatment.

Each removes a specific kind of confounding, and each rests on an assumption you must argue for rather than test. That is the honest position, and stating the assumption is the professional standard rather than an admission of weakness.

Identification comes before estimation. The prior question is whether the causal quantity can be recovered from the available data at all. If it cannot, no model, no volume of data and no compute budget will help, and the appropriate output is that the question cannot be answered with what you have. This is the part most often skipped, and skipping it produces confident numbers that mean nothing.

Run sensitivity analysis. Rather than asserting no unmeasured confounding, quantify how strong such a confounder would have to be to overturn your conclusion. If the answer is "implausibly strong", you have a defensible finding. If it is "about as strong as the variables you did measure", you have a hypothesis.

Can language models do this?

A reasonable question given where the field's attention is, and the answer is contested in an interesting way.

Benchmarks now exist specifically to test causal reasoning across Pearl's three rungs rather than pattern-matching on causal vocabulary. Results are mixed and depend heavily on how the question is posed. Models do well when the causal structure is stated in the prompt and the task is applying the rules. They do considerably less well when the structure has to be inferred, and they are susceptible to surface cues, agreeing that a relationship is causal when the phrasing sounds causal.

There is a deeper reason for scepticism. These models are trained to predict text, which is a rung-one operation over a corpus of human writing. Human writing contains a great deal of correct causal reasoning, so a model that predicts it well will reproduce correct causal claims about situations that appeared in its training data. Whether that constitutes causal reasoning or a good imitation of it is exactly the disagreement, and it is not resolvable by looking at outputs, since both accounts predict the same outputs on familiar cases.

The practical position: a model is useful for surfacing candidate confounders you had not considered, which is a real contribution to a hard task. It is not a substitute for the identification argument, because that argument depends on knowledge of how your data came to exist, which is not in the model.

What is unresolved

Whether causal structure can be learned from observational data. Algorithms exist that recover graphs under conditions including no unmeasured confounders and faithfulness. Those conditions cannot generally be verified in practice, which makes the guarantees difficult to rely on. Whether this is a temporary state or a fundamental limit remains open.

Whether prediction and causation can be unified. The majority view treats them as different questions requiring different tools. A minority position holds that a sufficiently good model of a system implies its causal structure and that the separation is an artefact of current methods. The evidence has not settled it, and the answer would change how the whole field is taught.

How to evaluate causal claims at scale. Prediction has held-out accuracy, which is why it industrialised. Causal inference has no equivalent, which is a large part of why it has not. Whether a workable proxy exists, or whether careful case-by-case argument is irreducible, determines whether this ever becomes routine practice rather than specialist work.

The counter-argument

A piece insisting on causal rigour owes an account of when the insistence is misplaced, because it frequently is.

Most machine learning does not need it. A spam filter, a recommendation engine, a demand forecast, a fraud score, an image classifier: none of these involves intervening on the thing being predicted. Prediction is the correct target, association is the right rung, and adding causal machinery would cost time and buy nothing.

The methods also have real costs. They require assumptions that cannot be tested, expertise that is scarce, and time that competes with shipping. A team that adopts causal inference for everything will move slowly and will still be wrong sometimes, because the assumptions can be wrong. There is a version of causal rigour that is mostly ceremony, producing an impressive-looking identification argument around an estimate no more reliable than the naive one.

And randomisation is not always available. Some interventions cannot be assigned at random for ethical reasons, some for practical ones, and some because the unit of treatment is too large. Telling those teams to run an experiment is advice they cannot use.

The defensible position is narrower than the strong one. When the model output will change a decision about how to act, the question is causal and prediction accuracy does not answer it. When nothing will be intervened upon, association is sufficient and the rest is overhead.

The short version

Machine learning models answer what tends to occur together. Decisions ask what happens if you act. These are different questions with different answers, and no accuracy metric reveals when you have substituted one for the other. A pneumonia model that found asthma patients at lower mortality risk was correct about its data and would have killed people in deployment, because the lower risk reflected the aggressive treatment those patients already received.

Pearl's ladder names the distinction: association, intervention, counterfactual. Standard models sit on the first rung; decisions live on the second. The gap is not closed by more data or higher accuracy. The most common attempted fix, adjusting for every available variable, actively introduces bias when it conditions on a collider, and selection into a dataset is a collider whenever selection depends on both treatment and outcome, which is often.

The gap between knowing this and doing it is documented and large. A review of ninety machine learning studies on immune checkpoint inhibitors found none using causal inference, and thirty-six melanoma studies produced the same count. The reasons are structural: prediction has a held-out accuracy number and causation has no ground truth, so one industrialised and the other did not.

Randomisation answers the question directly and an experiment on a few thousand users beats observational data with millions of rows. Where randomisation is impossible, difference-in-differences, instrumental variables, regression discontinuity and propensity methods each remove a specific confounding structure at the cost of an assumption you must argue rather than test. Identification, meaning whether the quantity is recoverable at all, comes before estimation and is the step most often skipped.

The rule worth keeping is that the moment a model's output will change what someone does, accuracy has stopped being the relevant measure, and nothing in the standard toolkit will tell you that the switch has happened.

Common questions

What is the difference between correlation and causation in machine learning? Correlation is what tends to occur together in observed data, which is what every standard model learns. Causation is what would happen if you intervened, which is what almost every decision requires. The distinction matters because observed data records a world in which decisions were already being made, so a model learns the effects of existing policy alongside everything else, with no marker separating them. The pneumonia case is the standard example: asthma appeared protective because asthmatic patients were already being treated aggressively.

Why can an accurate model still produce bad decisions? Because accuracy measures how well the model predicts what did happen, and a decision asks what would happen under a change the data does not contain. A churn model can identify precisely who will leave and say nothing about who would respond to a retention offer. A pricing model can find higher prices correlated with higher revenue in a segment that was charged more because it was less price-sensitive. In each case the model is right and the action based on it underperforms, with no error visible anywhere.

What is Pearl's ladder of causation? Three levels of question requiring different tools. Association, what tends to occur together, where all standard machine learning sits. Intervention, what happens if I act, formalised with the do-operator, which distinguishes observing a treatment from administering it. Counterfactual, what would have happened otherwise, which is hardest because the alternative was never observed. Most costly errors occur in the gap between the first two rungs.

Should I control for every variable I have? No, and this is the most common mistake because it feels careful. Conditioning on a collider, a variable influenced by both treatment and outcome, creates an association that does not otherwise exist. Selection into a dataset is a collider whenever selection depends on both, so studies of hospitalised patients, active users or successful projects all carry it. Which variables to adjust for is a structural question requiring a hypothesis about how the data came to exist, and no procedure reads the right answer off the data.

How do I actually estimate a causal effect? Randomise if you can, which is what an A/B test does, and the value lies in the random assignment rather than the subsequent statistics. When randomisation is impossible: difference-in-differences compares changes over time between exposed and unexposed groups; instrumental variables use something affecting treatment but not the outcome directly; regression discontinuity compares units just above and below an arbitrary threshold; propensity methods construct comparable groups. Each rests on an assumption you must argue for rather than test.

What is identification and why does it come first? Whether the causal quantity can be recovered from the available data at all, which is separate from and prior to estimating it. If a quantity is not identified, no model, no volume of data and no compute resolves it, and the correct output is that the question cannot be answered with what you have. Skipping this step produces confident estimates that mean nothing, and it is the step most often skipped.

Can large language models reason causally? Contested. On benchmarks testing the three rungs, models perform reasonably when the causal structure is supplied in the prompt and much less well when it must be inferred, and they are susceptible to surface cues that make a relationship sound causal. There is a structural reason for caution: predicting text is itself an associational operation over a corpus containing correct causal reasoning, so reproducing correct claims about familiar situations does not distinguish reasoning from imitation. Useful for surfacing candidate confounders; not a substitute for the identification argument, which depends on knowing how your data came to exist.

When do I not need causal inference? Whenever nothing will be intervened upon. Spam filtering, recommendation, demand forecasting, fraud scoring and image classification are all prediction problems where association is the right rung and causal machinery would cost time for no benefit. The line is whether the model's output will change a decision about how to act. If it will, the question is causal. If not, accuracy is the right measure and the rest is overhead.