“I think Chollet's "On the Measure of Intelligence" paper[1] is the best writing I've seen on this”
On the Measure of Intelligence
把智能定义为技能获取效率,并提出 ARC 基准来衡量
Chollet 回顾心理学和 AI 中定义与衡量智能的方法,指出只看某项任务上的技能会被先验知识和海量训练数据掩盖系统自身的泛化能力。他基于算法信息论提出把智能看作技能获取效率的形式化定义,给出通用 AI 基准应遵循的准则,并推出 Abstraction and Reasoning Corpus(ARC)基准。
HN 上的人在讨论 AI 是否真的有智能、基准测试是否有意义时反复引用它,称它是对这一问题写得最好的文章,并常把它和 ARC 基准一起提。
学习路径
读这篇之前,按顺序读书架上的 3 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出Chollet 批评深度学习只在局部泛化、容易被表层统计规律欺骗,这条路径先补上「脆弱性」和「类人智能」两方面的背景,再读他对智能的重新定义。
- 1
对抗样本说明网络的泛化很脆弱,是 Chollet 质疑只看任务表现的背景之一。
读读对抗样本部分:微小扰动如何让分类器出错;单元分析部分可略读。
- 2
显示 CNN 倾向抓表层统计规律而非高层抽象,对应目标论文区分「技能」与「泛化能力」的动机。
读抓傅里叶滤波实验的思路与结论:网络学到的是什么、没学到什么。
- 3
认知科学视角的类人 AI 主张(核心知识、因果模型、组合性),Chollet 的先验与核心知识概念与此一脉相承。
读读关于核心能力(直觉物理、直觉心理)和「学得快」的论述;各子领域细节可跳过。
- 4On the Measure of Intelligence
把智能定义为技能获取效率,并提出 ARC 基准。
读抓智能定义的形式化(先验、经验、泛化难度)与 ARC 的设计思路。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“a wonderful paper (which I probably found here on HN) on the subject”
“is the best overview of the challenges I've read. Highly recommended.”
“Francois Chollet has a good recent paper about this [1].”
“I agree with him. We should move away from benchmarks that involve training and evaluating algorithms on the same datasets.”
“There is a nice paper on this topic.”
看全部 20 条 →
“This is a great meta paper questioning the goal of the field itself, and proposing ways to formally evaluate intelligence in a computational sense.”
“I recommend "On the Measure of Intelligence" by François Chollet”
“You know, François Chollet literally did this, and people don't listen. People should listen to Chollet more.”
“Start at page 48 from this paper to see lots of cool tests, totally trivial for us humans to solve, that apparently AI have issues with.”
“I found it very useful for thinking about what it means for an AI to be generally intelligent.”
“You should go back and read the original ARC-AGI paper to see what this is about+.”
“The Cholet paper is a good attempt”
“I rather like this paper, but I think it is generous to say their benchmarks measure "intelligence".”
“ARC-AGI is a laudable effort, but its fundamental premise is indeed debatable:”
“Which makes these benchmarks mostly post hoc rationalizations.”
“Francois Chollet tries to address some of these problems by tackling how intelligence is measured.”
“it (also from Chollet) [2] makes a strong claim about the necessity of”
“This reminds me of The Abstraction and Reasoning Corpus [1] by François Chollet, accompanying his paper On the Measure of Intelligence [2]”
“a recent attempt at designing a psychometric test encoding human cognitive priors: On the Measure of Intelligence - the ARC dataset.”