Home/Blog/Evaluation & evidence/Generation takes seconds. Debunking takes hours.

Generation takes seconds. Debunking takes hours.

Four open source projects closed their doors in one month. The cause is not bad contributions but a cost ratio that inverted, and the same maintainer who shut his bounty credits the technology with finding 100 real bugs.

TL;DR. In January 2026 Daniel Stenberg closed curl's six-year bug bounty, which had paid out over $100,000 across 87 confirmed vulnerabilities. The valid rate had fallen from roughly one in six to 5%, with 20% of submissions machine-generated. In the same month Ghostty began closing drive-by generated pull requests, tldraw moved to auto-closing all external pull requests, and Jazzband, which maintained 84 Python projects, announced it was shutting down, reporting that 1 in 10 generated pull requests met project standards. The mechanism is a cost ratio, not a quality problem. Producing a plausible report takes seconds; reproducing and refuting one takes hours, and the refuting is done by volunteers. And the same maintainer who closed the bounty later credited machine-assisted analysis with surfacing more than 100 real curl bugs that fuzzing, static analysis and multiple human audits had missed.

---

Status: established. Sources are primary where possible: Stenberg's own posts announcing and explaining the closure, the Jazzband announcement, and repository policy changes at Ghostty and tldraw. The Stack Overflow and Tailwind figures are secondary and are labelled where used.

---

What closed, and when

January 2026, curl. Stenberg closed the HackerOne bug bounty after six years, 87 confirmed vulnerabilities and more than $100,000 paid. His October 2024 figures showed roughly one in six reports clearing as a valid CVE. By 2025 that had fallen to 5%, with 20% of submissions machine-generated.

January 2026, Ghostty. Mitchell Hashimoto declared drive-by generated pull requests would be closed without discussion. Hashimoto uses these tools heavily in his own work, which is what makes the position informative rather than reflexive.

January 2026, tldraw. Steve Ruiz configured the repository to auto-close all external pull requests. Not generated ones. All of them.

March 2026, Jazzband. Jannis Leidel announced the organisation, which collectively maintained 84 Python projects, was sunsetting. The reported figure: 1 in 10 generated pull requests met project standards.

Four projects, one quarter, and the responses escalate from filtering to disclosure requirements to closing the door entirely.

The mechanism is arithmetic

A generated pull request looks right. Coherent commit message, correct files touched, a legitimate-sounding problem described. The defects are in the logic, which is only visible after reading it.

So every submission costs a human review regardless of quality, and that is the whole problem.

Producing one takes seconds. Evaluating one takes an hour or more. At a 1-in-10 acceptance rate, obtaining one usable contribution costs ten reviews.

The bounty case is starker. A confident, well-formatted, entirely fabricated vulnerability report has to be read, reproduced, and refused in writing. Multiply by hundreds and a programme designed to find bugs becomes a denial-of-service attack on the person running it.

None of this requires the contributions to be malicious or even bad. It requires only that checking costs more than producing, and that the checker is unpaid.

Which is the structure underneath this whole territory

Research fraud doubles every 1.5 years while retraction doubles every 3.3, so the correction loses by construction.

A generated page costs nothing to produce and passes every automated quality check, because the checks measure delivery and assessing worth is the expensive part.

Denying authentic evidence takes a sentence while fabricating it takes effort, so the cheap move is the damaging one.

And here, generation takes seconds and refutation takes hours.

Four subjects, one shape: the cost of producing a claim fell to near zero and the cost of checking it did not move. Every system in this territory was built when those two costs were comparable, and none of them was designed with the ratio written down.

The part that complicates the story

Stenberg is not an opponent of the technology, and his own record says so.

In October 2025 he documented that researcher Joshua Rogers, using machine-assisted analysis tools, surfaced more than 100 real curl bugs that had survived years of aggressive fuzzing, compiler flags, static analysis and multiple human security audits.

Those are bugs nothing else found, in one of the most heavily scrutinised codebases in existence.

The difference is one step. Rogers filtered the output through his own expertise before submitting anything. The bounty submissions did not.

So the problem is not generated content. It is unverified content submitted by someone who has not paid the verification cost, which transfers that cost to a volunteer who did not consent to it.

That distinction is the whole finding, and it is why "ban the tools" and "the tools are fine" are both wrong.

What the responses have in common

The measures being adopted all raise the cost of submitting rather than trying to detect generation.

curl now requires a reproducible test case, which unverified reports typically cannot supply. Others require disclosure of assistance, participation in the issue thread before a pull request, or contribution history that takes time to accumulate.

None of these is a detector, which is consistent with what detection is worth: false positive rates that land on the wrong people, and evasion that costs a paraphrase.

They are friction priced to make bulk submission unprofitable while remaining trivial for a genuine contributor. A person who actually reproduced the bug already has the test case.

The second-order effect

A separate mechanism is being modelled, and it is slower and harder to see.

Research from Central European University and the Kiel Institute models delegating package selection to agents that do not read documentation, file bugs or engage maintainers. Where a project's returns depend on that engagement, the model predicts a negative feedback loop: fewer documentation visits, fewer human bug reports, eroded maintainer incentives, and declining software availability and quality despite productivity gains.

Two reported data points fit the shape. Stack Overflow activity fell about 25% within six months of ChatGPT's launch. Tailwind CSS downloads climbed while documentation traffic fell 40% and revenue dropped 80%.

Both are secondary figures and neither establishes causation. They are consistent with the model and they are not a test of it.

The reason this matters more than the pull request flood is that the flood is visible and generates responses, while this one shows up as usage rising and everything else quietly falling away.

Three things this establishes

A cost ratio can break a system with no bad actors in it. Nobody in the curl story is a villain. Contributors wanted bounties, the tools produced plausible output, and the maintainer was overwhelmed by arithmetic.

Friction beats detection where the asymmetry is the problem. Requiring a reproducible test case does not classify anything. It moves the verification cost back to whoever is proposing the claim, which is the only intervention that addresses the actual mechanism.

And the same technology sits on both sides. More than 100 real bugs found by machine-assisted analysis, in the same project, in the same period, by someone who verified before submitting. Any account that treats this as a story about the tools has to explain that.

What it does not establish

That open source is collapsing. Four projects is four projects, against an estimated 1.4 million maintainers. High-profile closures are visible; the distribution is not.

That the closures are permanent. These are policy responses to a volume shock, and policies change.

That the vibe coding model is correct. It is a model, its predictions are not yet tested, and the supporting figures are correlational.

And nothing about any individual contributor. Every figure here is aggregate, and the maintainers quoted are describing volume rather than accusing anyone.

What is unresolved

Whether friction requirements hold. A reproducible test case is a real barrier today. It is not obviously one in two years.

How the burden distributes below the visible projects. curl and tldraw have profile. The typical maintainer has none, and nobody is measuring what happens to them.

Whether the engagement loop is real or modelled. Documentation traffic down 40% while downloads rise is suggestive, and one project is not evidence.

And whether the verified-submission norm can be enforced. Rogers did the work before submitting. Nothing makes that the default except culture, which is what the closures are trying to defend.

The counter-argument

Maintainer burnout predates all of this. Unpaid people carrying critical infrastructure was a crisis in 2014, and attributing a long-running structural problem to a recent technology overstates its role. The volume shock may be the trigger rather than the cause.

Auto-closing all external pull requests is not a proportionate response to generated ones. tldraw's policy excludes every outside contributor, including the ones the model is supposed to be protecting, which suggests the burden was already unmanageable and this was the occasion rather than the reason.

The valid-rate collapse has an alternative reading. A bounty that pays for vulnerabilities attracts volume when submission becomes cheap, and a falling valid rate is what a lowered barrier produces in any incentive scheme. That is a bounty design problem as much as a technology problem, and bounties have had this failure mode before.

And four closures is a small sample chosen for being newsworthy. Projects that adapted quietly do not generate coverage, which is exactly the survivorship problem this corpus applies elsewhere and should apply here.

The short version

January 2026: curl closed its bug bounty after six years, 87 confirmed vulnerabilities and over $100,000 paid, with the valid rate down from about one in six to 5% and 20% of submissions machine-generated. The same month Ghostty began closing drive-by generated pull requests and tldraw moved to auto-closing all external pull requests. In March, Jazzband, maintaining 84 Python projects, shut down, reporting 1 in 10 generated pull requests met standards.

The mechanism is a cost ratio. A plausible submission takes seconds to produce and an hour or more to evaluate, and the evaluation is done by volunteers. At a 1-in-10 acceptance rate, one usable contribution costs ten reviews. A programme built to find bugs becomes a denial of service on the person running it.

Which is the shape underneath this entire territory. Fraud doubling twice as fast as retraction. Generated pages passing every delivery metric. Denial costing a sentence where fabrication costs effort. Producing a claim got cheap and checking one did not.

And the complication is in the same project. Stenberg later credited machine-assisted analysis with surfacing more than 100 real curl bugs that fuzzing, static analysis and multiple audits had missed. The researcher filtered the output through his own expertise before submitting. The bounty flood did not.

So the responses target the cost, not the origin: reproducible test cases, disclosure, prior participation, accumulated history. None of them detects anything. They move the verification cost back to whoever is making the claim, which is the only place it was ever supposed to sit.

Common questions

What actually happened in January 2026? Daniel Stenberg closed curl's HackerOne bug bounty after six years, 87 confirmed vulnerabilities and more than $100,000 paid out. The valid rate had fallen from roughly one in six reports clearing as a CVE in late 2024 to about 5%, with 20% of submissions machine-generated. In the same month Mitchell Hashimoto said drive-by generated pull requests to Ghostty would be closed without discussion, and Steve Ruiz set tldraw to auto-close all external pull requests. In March, Jazzband, which maintained 84 Python projects, announced it was sunsetting, reporting that 1 in 10 generated pull requests met project standards.

Why is volume the problem rather than quality? Because a generated submission looks correct until read. Coherent commit message, right files touched, a plausible problem described, with the defects in the logic. That means every submission costs a human review regardless of whether it is any good. Producing one takes seconds and evaluating one takes an hour or more, so at a 1-in-10 acceptance rate a single usable contribution costs ten reviews, all performed by unpaid volunteers.

Is this an argument against using AI in open source? No, and the strongest evidence against that reading comes from the same maintainer. In October 2025 Stenberg documented that researcher Joshua Rogers, using machine-assisted analysis tools, surfaced more than 100 real curl bugs that had survived years of fuzzing, compiler flags, static analysis and multiple human security audits. The difference is that Rogers filtered the output through his own expertise before submitting. The problem is unverified output submitted by someone who has not paid the verification cost.

What are projects doing instead of banning the tools? Raising the cost of submitting. curl requires a reproducible test case, which unverified reports typically cannot supply. Others require disclosure of AI assistance, participation in the issue thread before opening a pull request, or contribution history that takes time to build. None of these detects anything, which is deliberate: they are friction priced to make bulk submission unprofitable while staying trivial for someone who genuinely did the work and already has the test case.

What is the second-order effect? A modelled one, slower and harder to see. Research from Central European University and the Kiel Institute models developers delegating package selection to agents that do not read documentation, file bugs or engage with maintainers. Where project returns depend on that engagement, the model predicts fewer documentation visits, fewer human bug reports, eroded maintainer incentives and declining software quality despite productivity gains. Two reported figures fit: Stack Overflow activity falling about 25% within six months of ChatGPT's launch, and Tailwind CSS downloads rising while documentation traffic fell 40% and revenue dropped 80%. Both are secondary and correlational.

Does this mean open source is collapsing? No. Four projects is four projects against an estimated 1.4 million maintainers, and closures generate coverage while quiet adaptation does not, which is a survivorship problem. What the closures establish is that the cost ratio can break a well-run project with no bad actors involved, not that it has broken the ecosystem.

What is the strongest objection to this framing? That maintainer burnout long predates generative tools. Unpaid people carrying critical infrastructure was already a crisis a decade ago, so the volume shock may be the trigger rather than the cause. A related objection is that tldraw's response, closing all external pull requests including human ones, suggests the burden was already unmanageable and this was the occasion rather than the reason. There is also a bounty-design reading: any incentive scheme sees its valid rate fall when the cost of submitting drops.

What is the general lesson? That systems built when producing a claim and checking it cost roughly the same amount can break when only one of those costs falls. Nothing in the curl story requires a villain. It requires only that checking costs more than producing and that the checker is unpaid, which describes peer review, bug bounties, moderation queues and a great deal else.

Sources

Primary documents only. Where a claim rests on a single report, the entry says so.

  1. The end of the curl bug-bounty Daniel Stenberg, 26 January 2026 The primary account: six years, 87 confirmed vulnerabilities, over $100,000 paid, the valid rate falling from roughly one in six to 5%, and 20% of submissions machine-generated.
  2. A new breed of analyzers Daniel Stenberg, October 2025 The counterpoint from the same maintainer: machine-assisted analysis surfacing more than 100 real curl bugs that fuzzing, static analysis and multiple human audits had missed.
  3. How OSS Contribution Policies Changed in Response to AI Slop Codenote The collected policy responses at curl, Ghostty, tldraw and Jazzband, with dates and repository changes, and the framing of generation and debunking costs having inverted.
  4. AI Vibe Coding Threatens Open Source as Maintainers Face Crisis InfoQ The Central European University and Kiel Institute model of delegated package selection eroding maintainer incentives, and the secondary Stack Overflow and Tailwind figures used here with that label.

Further reading

The primary literature behind the claims above, drawn from the concept entries this post links to, so a claim carries the same source here as it does there.

  • Wong et al. (2021), External Validation of a Widely Implemented Proprietary Sepsis Prediction Model — a failure mode that produces no artefact and is therefore invisible to any register. :: https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307 Survivorship Bias
  • Pineau et al., reproducibility programme work, and Bouthillier et al. (2021), Accounting for Variance in Machine Learning Benchmarks — why a literature of reported results describes what was submitted. :: https://arxiv.org/abs/2103.03098 Survivorship Bias

Learn the concepts

← All posts