“The study you referred to sounded super interesting, so I looked it up to read later.”
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
随机对照试验:AI 工具使资深开发者慢了 19%
研究让 16 名资深开源开发者在自己平均有 5 年经验的成熟项目里完成 246 个任务,每个任务随机分配为允许或禁止使用 2025 年初的 AI 工具(主要是 Cursor Pro 与 Claude 3.5/3.7 Sonnet)。开发者事前预测 AI 会让耗时减少 24%,事后仍估计减少了 20%,但实测允许使用 AI 反而使完成时间增加了 19%,与经济学和机器学习专家的预测也相反。
评论者反复引用它说明开发者对 AI 提速的主观感受并不可靠,也有人批评样本只有 16 人、对象是早期 AI,结论已经过时。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“this should be studied more in depth over a longer time period since the current study had a 50 hour threshold”
“Folks probably should be skeptical of a study with the combination of a small number of subjects with a broad claim.”
“The METR paper surveyed just 16 developers to arrive at their conclusion. Not sure how that got past review.”
“it seems a bit silly to rely on a paper about early-2025 ai when people are using late-2026 ai”
“And that self-assessment is flawed.”
看全部 20 条 →
“(even when they believe that it does)”
“There was a recent study concluding that AI made experienced developers 20% SLOWER to complete tasks rather than any faster !”
“I can tell this is going to be the most misquoted study in blogs and pop-sci books after the 10,000-hour mastery study. And it's just a preprint!”
“The study you are alluding to is this one by METR (Model Evaluation & Threat Research):”
“There's literally a study out the shows when developers think LLMs are making them 20% faster”
“this is what Metr found in its study of developer productivity.”
“Unlike higher level programing languages AI doesn't actually make programmers more efficient.”
“there is explicit evidence it does slow them down, even when they think it sped them up: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”
“that METR study should have completely killed any illusions that that is a reliable metric”
“The METR study referenced (but not linked) in that paragraph has a breakdown of time spent Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity in figure 19 on page 33 which contradicts the description in the Register article.”
“The following paper backs this idea Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”
“Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down.”
“But I know that using in AI in software engineering reduces productivity by almost 20%.”
“A double-difference comparison between GitHub projects that have adopted Cursor and paired projects that have not:”