Nornic Register · Living document
“AI makes experienced developers 19% slower.”
A real result from sixteen developers, still circulating after its own authors withdrew the larger follow-up as unreliable.
The source
- Study
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- Publisher
- METR
- Publication type
- Randomised controlled trial, published research
- Sample
- 16 developers, 246 tasks
- Fieldwork
- February – June 2025
- Funding & conflicts
- METR — a non-profit AI evaluation organisation
- Verified on
- 2026-07-26
The arithmetic
The study is careful and its authors are unusually candid about its limits. The problem is not the study; it is the citation.
n = 16. That supports a directional observation about a specific population — experienced open-source maintainers working in repositories they already know well. It does not support a general claim about developers.
In February 2026 METR abandoned its own 57-developer follow-up, stating it was unreliable due to selection bias. The team noted the raw data pointed toward a speedup and wrote that the true effect "could be much higher".
Almost nobody citing the 19% figure cites that correction. The original number now travels without the authors' own revision attached to it.
What it does support: that experienced developers can be slower than they expect on familiar code, and that self-reported speedup is unreliable. That is a genuinely useful finding, and a smaller one.
What would change this verdict
A replication at meaningful scale, with the selection-bias problem addressed, reporting a negative effect. METR's own withdrawn follow-up suggests the opposite is more likely.
Sources