批评namaria · 2025-04
“As usual with these bombastic blog posts, you have to follow the paper trail to find the caveats that deflate the whole thing.”
用人类耗时衡量 AI 能力:50% 成功的任务时长约每七个月翻倍
论文提出“50% 任务完成时间跨度”指标,即 AI 能以 50% 成功率完成的任务所对应的人类耗时,并在 RE-Bench、HCAST 和 66 个新短任务上为人类专家计时。结果显示 Claude 3.7 Sonnet 的时间跨度约 50 分钟,自 2019 年以来大致每七个月翻倍,增长主要来自可靠性、纠错、逻辑推理和工具使用的提升。作者也讨论了结果的外部效度等局限。
评论者把它当作“AI 能力呈指数增长、时间跨度约每七个月翻倍”的引用来源,也有人批评相关博客式宣传淡化了论文里的局限。
“As usual with these bombastic blog posts, you have to follow the paper trail to find the caveats that deflate the whole thing.”
“As the report models, this time horizon doubles approximately every 7 months (R² = 0.98 on a log-scale)”
“the METR study showed the rate of change is t doubles every 7 months”
“Competence is increasing exponentially[1]”
“Mostly it wasn't estimated, but rather measured:”