METR developer RCT
The 2025 randomised controlled trial in which 16 experienced open-source developers were 19% slower with AI tools on real issues while believing they were 20% faster.
METR randomised 246 real tasks on the developers' own mature repositories (average 22k+ stars, 1M+ lines) to AI-allowed or AI-disallowed, and screen-recorded them. Measured completion time rose 19%; developers had forecast a 24% speed-up and still believed in a 20% speed-up afterwards. Experts predicted 38–39% speed-ups. The study is the strongest evidence for a perception gap and for the conditions (deep familiarity, large codebases, high quality bar) under which early-2025 tools did not help.