61% flagged for non-native writers, 3% for native

TL;DR. A 2023 study published in Patterns tested seven major AI text detectors on TOEFL essays written by non-native English speakers. It found a 61.3% false positive rate on that set against roughly 3% on native-speaker essays. The mechanism is not a tuning error: perplexity-based detection reads simple, predictable vocabulary as a machine signal, and non-native writing has lower perplexity because it has less linguistic variability. OpenAI launched its own classifier in January 2023 and withdrew it that July, disclosing it identified 26% of AI-generated text while wrongly flagging 9% of human text. Adversarial work finds paraphrasing cuts detection rates by an average of 87.88%. Students have been suspended on detector scores and have sued, and several universities have disabled the tools.

---

Status: established, with active vendor dispute. The primary source is Liang and colleagues in Patterns, 2023. Vendor accuracy claims are almost entirely self-reported and are labelled as such. Where a vendor disputes an independent finding, both are given.

---

The study

Liang and colleagues tested seven widely used AI text detectors on a benchmark of 91 TOEFL essays written by non-native English speakers, alongside essays by native speakers.

On native-speaker essays the detectors were nearly perfect, with a false positive rate around 3.2%.

On the TOEFL essays the average false positive rate was 61.3%. Roughly two in three legitimate essays were classified as machine-written.

And the explanation is mechanical. Most detectors work on perplexity, a measure of how predictable a text is to a language model. Machine-generated text tends to be low-perplexity because generation favours likely continuations. Writing by someone working in a second language also tends to be low-perplexity, because the vocabulary is narrower and the constructions more conventional.

The detector is not mistaking non-native writing for machine writing by accident. It is measuring a property the two genuinely share, which is why this is a construct problem rather than a calibration one.

Later work reports that neurodivergent students are also flagged at elevated rates, for a related reason: repeated phrasing and consistent vocabulary read as low-variability text.

The vendor's own withdrawal

OpenAI released an AI text classifier in January 2023 and shut it down in July of the same year.

Its disclosed performance: it correctly identified 26% of AI-generated text, and wrongly flagged 9% of human text.

That is a company withdrawing its own product on published accuracy grounds, which is unusual and is the single most informative event in this subject. The organisation with the best possible access to how its models write could not build a reliable detector of them.

And paraphrasing removes what is left

Adversarial evaluation finds detection rates falling by an average of 87.88% after targeted paraphrasing. One widely cited method drops from 70.3% to 4.6% accuracy after basic paraphrasing alone.

Which means the tools are weakest exactly where the stakes are highest. Someone deliberately concealing machine assistance paraphrases. Someone writing honestly in a second language does not, and gets flagged.

The error profile is therefore inverted relative to the purpose: high false positives on the innocent, high false negatives on the deliberate.

What the vendors say

Fairly, because they dispute this and some of the dispute is reasonable.

Turnitin states a document-level false positive rate below 1%, and critiques the Stanford study as resting on 91 works of under 150 words, reporting no comparable bias on larger databases of longer text. It is worth noting that the Stanford study did not test Turnitin. Independent readings put its sentence-level rate nearer 4%, and Turnitin itself acknowledges a variance of plus or minus 15 percentage points on its scores, which means a 50% result could legitimately be anywhere from 35% to 65%.

GPTZero published a response titled arguing that ESL bias in AI detection is an outdated narrative, alongside an updated model. Independent evaluation of that updated model reported a 7.7% false positive rate on the same TOEFL benchmark, or 1.1% under a more generous labelling.

One newer detector reports 0% false positives on the TOEFL benchmark, having held it out of training, and roughly 0.01% overall on first-party testing. University of Chicago Booth research in 2025 found it essentially zero across passage lengths, with GPTZero and one competitor near 1% and open-source tools failing badly.

So the field has improved, and the improvement is measured largely by vendors on the benchmark that exposed the problem. That is the external validation question again: better numbers reported by the party selling the product, on a test set now known to everyone building for it.

What it costs when it is wrong

A false negative is a policy nuisance: one piece of undisclosed machine writing passes.

A false positive is an accusation of academic misconduct against a named person, with a disciplinary process attached.

Those are not comparable costs and the tools are optimised against a symmetric metric.

The consequences are documented. A student at Yale sued in February 2025 after being suspended following a detector flag, with the complaint citing discrimination against non-native English speakers. A student at the University of Michigan sued in 2026 after an accusation in which the instructor reportedly used AI-generated comparison text as evidence, a method that establishes only that a machine can produce something similar to the work in question.

Several institutions have responded by turning the tools off. Yale, UCLA, Berkeley, San Diego, Waterloo, Michigan State and Vanderbilt are among those reported to have disabled or restricted AI detection.

Which is why the prevalence question has no answer

The previous article noted that nobody knows what fraction of training data is machine-generated, and that detection is unreliable. This is the evidence for that.

Every published estimate of synthetic content prevalence rests on a detector. If the detector has a false positive rate that varies from 3% to 61% depending on who wrote the text, a prevalence estimate is a measurement of the detector applied to an unknown population mix, not a measurement of the population.

And the error is directional rather than noisy. Text by non-native writers is over-counted as synthetic; deliberately paraphrased machine text is under-counted. A prevalence figure inherits both, in proportions nobody can state.

Which makes the honest answer to how much of the web is AI-generated not a range but an absence.

Three things this establishes

A structural bias is not a bug to be patched. Perplexity-based detection measures predictability, and low predictability is a real property of both machine text and second-language writing. Improving the classifier does not separate two things that genuinely overlap on the measured dimension.

Withdrawal is stronger evidence than benchmarks. A company retiring its own detector on published figures says more than any vendor accuracy page, because it is a costly signal against interest.

And error asymmetry should determine deployment, not accuracy. A tool with 95% accuracy whose errors fall on a specific group, in a setting where a false positive is a disciplinary accusation, is not adequately described by 95%.

What it does not establish

That detection cannot improve. Independent research in 2025 found at least one detector with essentially zero false positives across passage lengths, which is a real result.

That the Stanford figure describes current tools. It tested 2023-era detectors on 91 short essays, and the field has moved, which is the vendors' strongest objection.

That every flag is wrong. Detectors do identify machine text at rates well above chance, and the false negative problem is as real as the false positive one.

And nothing about any individual case. The lawsuits described are proceedings whose outcomes are not settled here.

What is unresolved

Whether the improvement generalises. A detector reporting zero false positives on the benchmark that exposed the bias has been optimised in a world where that benchmark is public.

What happens under adversarial conditions at scale. Paraphrasing collapses detection in evaluation, and no deployed figure accounts for it.

Whether institutions can use these tools proportionately. Guidance now commonly warns against using a detector score as sole evidence, which raises the question of what it is evidence of.

And what the prevalence actually is. Nobody knows, and the instrument that would tell them is the subject of this article.

The counter-argument

Leading with a 2023 study on 91 short essays is unfair to 2026 tools. The field has improved measurably, at least one detector now reports essentially zero false positives on that exact benchmark under independent testing, and the article's framing borrows the rhetorical force of a finding that the vendors have partly addressed.

Turnitin's methodological objection is substantive. Ninety-one works under 150 words is a small sample of short texts, and short texts are where every detector performs worst. A study that did not test the most widely deployed tool is a weak basis for conclusions about that tool.

The lawsuits do not establish the technology is at fault. They establish that institutions used scores as evidence, which is a policy failure that would occur with any imperfect instrument, and guidance against sole-evidence use is the appropriate remedy rather than abandoning detection.

And the prevalence argument may prove too much. Every measurement instrument has error, and the standard applied here, that a directional error rate invalidates population estimates, would rule out a great deal of social measurement that is nonetheless useful with appropriate bounds.

The short version

Seven major AI detectors, tested on TOEFL essays by non-native English writers, produced a 61.3% false positive rate against roughly 3% on native-speaker essays. The mechanism is structural: perplexity-based detection reads predictable vocabulary as a machine signal, and second-language writing is predictable for reasons that have nothing to do with machines. Neurodivergent writers are reported to be flagged at elevated rates for a related reason.

OpenAI launched a classifier in January 2023 and withdrew it in July, disclosing that it caught 26% of machine text while flagging 9% of human text. The organisation with the best access to how its own models write could not build a reliable detector of them.

Adversarial paraphrasing cuts detection by an average of 87.88%, and one method falls from 70.3% to 4.6%. So the errors run backwards to the purpose: false positives on the honest, false negatives on the deliberate.

Vendors dispute this and have partly addressed it. One detector reports zero false positives on that benchmark under independent testing; another reports 7.7% after an update; a third disputes the study's sample and was not in it. The improvement is real and is largely measured on the test set that exposed the problem.

And the cost asymmetry is what matters. A false negative is a nuisance. A false positive is a misconduct accusation against a named person, and students have been suspended and have sued, while several universities have turned the tools off.

Which is why the prevalence question has no answer. Every estimate of how much text is machine-written rests on a detector whose error rate varies twentyfold with who wrote the text. That is a measurement of the instrument, not of the population.

Common questions

What did the Stanford study find? Liang and colleagues, publishing in Patterns in 2023, tested seven widely used AI text detectors on a benchmark of 91 TOEFL essays written by non-native English speakers. The average false positive rate was 61.3%, against roughly 3.2% on essays by native speakers. Nearly two in three legitimate non-native essays were classified as machine-written.

Why does that happen? Because most detectors measure perplexity, which is how predictable a text is to a language model. Machine-generated text tends to be low-perplexity because generation favours likely continuations. Writing by someone working in a second language also tends to be low-perplexity, because vocabulary is narrower and constructions more conventional. The detector is measuring a property the two genuinely share, which makes it a construct problem rather than a calibration error that tuning could remove.

Are other groups affected? Reported research indicates neurodivergent students are also flagged at elevated rates, for a related reason: repeated phrasing and consistent vocabulary produce the low-variability signal detectors read as machine-generated.

What happened to OpenAI's detector? It launched in January 2023 and was withdrawn in July 2023. The disclosed performance was that it correctly identified 26% of AI-generated text while wrongly flagging 9% of human text. A company retiring its own product on published accuracy grounds is a costly signal against interest, which makes it more informative than any vendor accuracy page.

Do detectors work against someone actively trying to evade them? Poorly. Adversarial evaluation finds detection rates falling by an average of 87.88% after targeted paraphrasing, with one widely cited method dropping from 70.3% to 4.6% accuracy after basic paraphrasing alone. The error profile therefore runs backwards to the purpose: high false positives on people writing honestly in a second language, high false negatives on people deliberately concealing machine assistance.

Have the tools improved? Yes, and the improvement is largely vendor-measured on the benchmark that exposed the problem. One detector reports zero false positives on the TOEFL set having held it out of training, and independent University of Chicago Booth research in 2025 found it essentially zero across passage lengths, with two competitors near 1%. Another vendor published an updated model whose independently evaluated false positive rate on the same benchmark was 7.7%. Turnitin, which the original study did not test, states a document-level rate below 1% while acknowledging plus or minus 15 percentage points of variance in its scores.

Why does the cost asymmetry matter more than accuracy? Because a false negative means one piece of undisclosed machine writing passes, which is a policy nuisance, while a false positive is an accusation of academic misconduct against a named person with a disciplinary process attached. Those are not comparable costs, and a tool optimised against a symmetric accuracy metric is not being evaluated on the thing that matters. Students have been suspended on detector scores and have filed suit, and several universities have disabled or restricted the tools.

Why does this mean nobody knows how much of the web is AI-generated? Because every prevalence estimate rests on a detector. If a detector's false positive rate varies from around 3% to around 61% depending on who wrote the text, then applying it to a population of unknown composition measures the detector rather than the population. The error is directional rather than noisy: non-native writing is over-counted as synthetic and deliberately paraphrased machine text is under-counted, and any prevalence figure inherits both in proportions nobody can state.