How to read an AI paper

Between 2017 and 2021 a group of researchers did to machine learning what reformers had done to psychology a decade earlier. They took a critical sample of the published literature, attempted to reproduce the central claims, and documented what went wrong.

The finding was not subtle. A meaningful fraction of state-of-the-art results, including in the highest-prestige subfields and the highest-prestige venues, could not be cleanly reproduced from the published artifact.

The reasons were structural rather than fraudulent. Papers did not release code. They did not specify the hyperparameters that produced the headline number. They did not report variance across random seeds. They compared new methods against weak or non-standard baselines. They did not disclose compute.

Every one of those is visible from the outside. You do not need to rerun an experiment to know whether a paper has given you enough to believe it, and the checks take about fifteen minutes once you know what they are.

Read in the wrong order deliberately

Papers are written to be persuasive in sequence and are best read out of sequence.

Title and abstract, two minutes. Answer three questions roughly: what problem, what approach, and is this relevant to something I actually care about. If the third answer is no, stop. Nobody reads every paper they open and pretending otherwise is how literature reviews become miserable.

Then the results tables, before the method. The claim lives in the numbers. Reading the method first primes you to find it convincing, and the method is usually more interesting than the evidence supporting it.

Then the baselines. Covered below, and this is where most claims fall.

Then the ablations, which tell you whether the interesting part of the method is the part doing the work.

Then the method, now that you know what it has to justify.

Then limitations, which in a good paper is the most informative section and in a weak one is three sentences of ritual.

The introduction and related work can be skipped on a first pass. They tell you what the authors want you to think the contribution is.

The four questions that settle most papers

What is the baseline, and is it fair?

This is the single highest-yield check and it fails constantly. A claimed improvement is a comparison, and a comparison is only as good as what it was against.

Three specific failures. The comparison omits a recent or stronger method, so the paper beats the state of the art from two years ago. The baselines were not tuned as carefully as the proposed method, which is nearly always true and rarely disclosed. Or the baseline is not stated at all, which turns a percentage into decoration.

Consider "improved accuracy by 40%". Forty percent better than what? Going from 50% to 70% is a 40% relative improvement, and on a binary task 70% may still be poor. A number without a baseline is not a result.

Which part of the method produced the gain?

That is what an ablation study answers, and its absence is a specific kind of gap. A paper proposing four changes and reporting one aggregate improvement has not shown that its interesting idea did anything. Frequently the gain traces to a mundane component, more data, a better learning rate schedule, longer training, and the novel mechanism contributes little.

If there are no ablations, the paper has demonstrated that the whole package beats the baseline and nothing about why.

How many times was this run?

A single number from a single run tells you almost nothing about a stochastic system. Cherry-picked seeds were named explicitly in the reproducibility literature as a failure mode, and the honest version reports mean and variance across multiple runs.

The check: if the paper reports a single figure with no variance, and the improvement is small, the result may be inside the noise. Ask what the seed-to-seed spread would have to be for the claimed gain to disappear, and whether that spread is plausible.

Could I rerun this?

Not whether you will, but whether you could. Is the code released. Are hyperparameters specified. Is the data available or described precisely enough to reconstruct. Is the compute disclosed, which matters because a result achievable only with resources nobody else has is a claim about a budget as much as a method.

A paper failing all four of these can still be correct. It has simply not given you grounds to believe it, which is a different property from being wrong and is the one you can assess.

A worked example, on a claim you will recognise

Take a hypothetical abstract of a shape that appears constantly, and walk the checks.

"We introduce a novel attention variant and demonstrate a 3.2 point improvement on a standard reasoning benchmark, achieving state-of-the-art performance."

Baseline. Three point two above what? The abstract does not say, which means the check has to move to the results table. There, the comparison is against a model from eighteen months earlier. A stronger method published four months ago is in the related work section and not in the table. The improvement over the actual state of the art is unstated and possibly negative.

Ablations. The method has three components: the attention variant, a modified position encoding, and a longer training schedule. The ablation table reports the full method and the full method minus the attention variant, a difference of 0.4 points. The remaining 2.8 came from somewhere and the paper does not say where.

Variance. A single number to one decimal place, no standard deviation, no mention of seeds. A 0.4 point effect on a benchmark of this size may be well inside seed variance, which would make the paper's central contribution unmeasurable.

Reproducibility. Code "will be released". Hyperparameters in an appendix, but only for the proposed method, not for the baselines, which is the asymmetry that makes the untuned-baseline problem invisible.

Verdict. Nothing here indicates the paper is wrong. It indicates that the headline number is attributable to the training schedule as easily as to the idea, that the state-of-the-art claim rests on an outdated comparison, and that the effect size for the actual contribution is within plausible noise.

That took under fifteen minutes and it changes what you would do with the paper. The finding is not "this is bad research". It is "this has not established the thing the title says it established", and those are different conclusions with different consequences.

What different papers owe you

The right checklist depends on what kind of claim is being made, and applying the wrong one produces unfair criticism.

A new-model paper owes you strong baselines, ablations, compute transparency and error analysis. The error analysis matters most and appears least: a paper reporting where its method fails is doing something the aggregate score cannot.

A benchmark paper owes you dataset construction detail, leakage control, annotation quality, a baseline suite, and a maintenance plan. That last one is routinely absent and determines whether the benchmark is useful in two years or saturated and abandoned.

A theory paper owes you clearly stated assumptions, a proof you can follow, and examples showing the result applies to something. The failure mode is a sound theorem about a setting nobody is in.

An applied paper owes you domain validity, deployment context, and evaluation matching the real decision. A model that scores well on a proxy metric while nobody establishes the proxy tracks the outcome has demonstrated something about the proxy.

A generative-model paper owes you contamination checks, task-specific evaluation, and stated limitations. Contamination checks are the specific obligation here because the training corpus may contain the evaluation.

Red flags, in order of how fast they should reduce trust

Critical, and disqualifying.

No baseline stated for a claimed improvement. The number means nothing without it.

Evaluation on the same data used for development. This is not a subtle error, and it appears more often than it should, usually through a held-out set that was consulted during tuning.

A metric introduced in the same paper that claims to improve on it. Possibly legitimate, and it requires an argument that the metric measures something independent of the method.

High, and usually decisive.

No ablations for a multi-part method. You cannot attribute the gain.

Single-seed results with a small improvement. The claim may be noise wearing a decimal point.

Untuned baselines. Almost universal and almost never disclosed. The question to ask is how much effort went into the comparison relative to the proposal.

A test set that is a subset of a public dataset the model may have trained on. Contamination, and specifically the kind that is checkable and usually not checked.

Medium, and worth noting.

Limitations section under four sentences. Not evidence of a problem and reliable evidence that nobody looked hard.

No compute disclosure. Prevents anyone assessing whether the comparison was fair in resources.

Results only on the benchmark the method was designed for. Generalisation is asserted rather than shown.

The critique test, which works in both directions

A useful heuristic borrowed from reviewing practice, and it applies to any assessment of a paper including your own.

A critique that never engages the specific evidence is not a critique. Weak reviews comment on the abstract and the discussion while never naming the table or figure that carries the claim. They demonstrate fluency with method names without judging whether the method fits the design.

The same test applies to praise. Someone recommending a paper who cannot say which result convinced them has responded to the framing rather than the evidence.

This gives you a fast check on secondary sources. When a summary, a thread or a news article describes a finding, ask whether it names the specific number and the comparison it was against. If not, the summariser did not read the evidence either, and you are now three steps from the data.

What a good paper looks like

Worth stating positively, because a list of failures makes everything look bad.

It compares against the strongest available method, tuned with visible effort. It ablates, and reports the case where the ablation was disappointing. It runs multiple seeds and reports the spread. It states where the method fails, not just where it works. Its limitations section is long enough to be uncomfortable. It discloses compute. It releases code that runs.

Papers like this exist and they are frequently less exciting than their neighbours, because honest reporting produces smaller headline numbers. A modest, well-evidenced improvement is worth more than a large one you cannot verify, and the field's incentives point the other way, which is why the checks are necessary.

What is unresolved

Whether checklists work. Reproducibility checklists were the structural response, adopted at major venues, requesting information about models, theoretical claims, data, code and experiments. Whether they improved reproducibility or added a compliance step to an already overburdened review process is disputed, and the evidence is mixed.

Whether frontier results can be assessed at all. The checks above assume a paper describing something reproducible in principle. Results from systems trained at costs beyond nearly every institution, on undisclosed data, are not assessable by these means. What replaces peer review under those conditions is not established, and the field is proceeding without an answer.

Whether reproducibility is the right target. Reproducing a result means getting the same number from the same artifact. Replicating means getting the same conclusion from an independent attempt, which is a stronger and rarer test. Most of the reform effort targets the first, which is easier to check and a weaker guarantee.

Whether automated review helps. Systems assessing papers are being deployed, and evaluations of automated research pipelines find substantial failure rates. Whether machine review raises the floor or adds fluent noise is open.

The counter-argument

Papers are not products. A conference paper is a progress report from an ongoing line of work, written under a deadline, and demanding production-grade evidence for every claim would slow the field considerably. Some tolerance for incomplete evidence is how research proceeds at all.

The reproducibility framing may overstate the problem. Failure to reproduce from a published artifact is not the same as a false result. Much of the gap traces to missing detail rather than incorrect findings, and the underlying claims frequently hold when the authors are asked.

Strong baselines are hard. Tuning a competitor as carefully as your own method requires expertise in someone else's system and an incentive structure that rewards making your comparison look worse. Criticising authors for imperfect baselines is fair and it is also asking for something the process does not support.

And this advice suits some readers and not others. A practitioner deciding whether to adopt a method needs these checks. Someone tracking a field's direction can read more loosely, because the aggregate signal across many papers is more robust than any single result. Applying maximum scrutiny to everything is its own inefficiency.

The short version

A sequence of studies between 2017 and 2021 found that a meaningful fraction of state-of-the-art machine learning results, including in the highest-prestige venues, could not be cleanly reproduced from the published artifact. The causes were structural: unreleased code, unspecified hyperparameters, no variance across seeds, weak or untuned baselines, and undisclosed compute. All of these are visible from outside the experiment.

Read out of order. Abstract, then results tables, then baselines, then ablations, then method, then limitations. The introduction tells you what the authors want the contribution to be.

Four questions settle most papers. What is the baseline and is it fair, since a number without a baseline is not a result and "improved by 40%" may describe going from 50% to 70% on a binary task. Which part of the method produced the gain, which is what ablations answer and which frequently traces to more data or longer training rather than the novel mechanism. How many times was this run, since a single seed on a stochastic system may report noise. And could this be rerun, meaning code, hyperparameters, data and compute.

Different papers owe different evidence. New-model papers owe baselines, ablations, compute and error analysis. Benchmark papers owe construction detail, leakage control and a maintenance plan. Applied papers owe evaluation matching the real decision. Generative-model papers owe contamination checks.

The heuristic that transfers furthest: a critique that never engages the specific evidence is not a critique, and neither is praise. When a summary describes a finding without naming the number and the comparison it was against, the summariser did not read the evidence either, and you are three steps from the data.

Common questions

How do I read a machine learning paper efficiently? Out of order. Spend two minutes on the title and abstract answering what problem, what approach, and whether it is relevant to you, and stop there if the answer to the third is no. Then read the results tables before the method, since the method is usually more interesting than the evidence for it and reading it first primes you to be convinced. Then baselines, ablations, method, and limitations. Introduction and related work can wait.

What is the biggest red flag in an AI paper? A claimed improvement with no stated baseline, or a baseline that is weak or untuned. A number is a comparison, and without knowing what it was compared against it carries no information. "Improved accuracy by 40%" may describe moving from 50% to 70% on a binary classification task, which is a large relative gain and possibly still poor absolute performance.

Why do ablation studies matter? Because they answer which part of a method produced the gain. A paper proposing four changes and reporting one aggregate improvement has shown that the package beats the baseline and nothing about why. The gain frequently traces to a mundane component such as more data, longer training or a better learning-rate schedule, while the novel mechanism contributes little. Without ablations you cannot attribute the result.

How can I tell if a result is statistically meaningful? Look for multiple runs and a reported spread. A single number from a single run of a stochastic system carries little information, and cherry-picked seeds were named explicitly in the reproducibility literature as a failure mode. If a paper reports one figure with no variance and the improvement is small, ask how large the seed-to-seed spread would need to be for the gain to vanish, and whether that is plausible.

What is the machine learning reproducibility crisis? A series of studies between 2017 and 2021 sampled published results, attempted to reproduce the central claims, and found that a meaningful fraction of state-of-the-art results in high-prestige venues could not be cleanly reproduced from the published artifact. The causes were structural rather than fraudulent: no code release, undocumented hyperparameters, no seed variance, weak baselines, undisclosed compute. Reproducibility checklists, code-sharing norms and replication challenges were the response.

What should a benchmark paper include that a model paper does not? Dataset construction detail, leakage control, annotation or generation quality, a baseline suite, and a maintenance plan. The maintenance plan is routinely absent and determines whether the benchmark remains useful or becomes saturated and abandoned. Model papers instead owe strong baselines, ablations, compute transparency and error analysis, with error analysis being the most informative and least common.

How do I evaluate a summary or news article about AI research? Ask whether it names the specific number and the comparison it was measured against. A critique or a recommendation that never engages the load-bearing table or figure has responded to framing rather than evidence, which is a documented pattern in weak reviews. If the summary omits the baseline, the summariser did not check it, and you are several steps removed from anything you could assess.

Are these checks fair to researchers? Partly, and the tension is real. Conference papers are progress reports written under deadline, not products, and demanding production-grade evidence for every claim would slow the field. Tuning a competing method as carefully as your own requires expertise in someone else's system and an incentive to make your comparison look worse. The checks are appropriate for deciding whether to adopt a method and excessive for tracking a field's direction, where the aggregate signal across many papers is more robust than any single result.