“See this cool this paper, "Theory of Mind May Have Spontaneously Emerged in Large Language Models" Evaluating Large Language Models in Theory of Mind Tasks”
Evaluating Large Language Models in Theory of Mind Tasks
用错误信念任务测 11 个 LLM 的心智理论能力
论文用自制的错误信念任务集评估 11 个大语言模型:共 40 个任务、640 条提示,每个任务包含一个错误信念场景、三个匹配的真实信念对照场景及其反转版本。较小较旧的模型一个任务也解不出,GPT-3-davinci-003 和 ChatGPT-3.5-turbo 解出 20%,2023 年 6 月的 ChatGPT-4 解出 75%,相当于六岁儿童的水平;作者探讨了这些结果的可能解释,包括心智理论可能作为语言能力提升的副产品自发涌现。
2023 年 GPT-4 刚发布时,评论者用它支持「LLM 具备基础心智理论、能做一定高阶推理」的看法,引用的多是这篇论文和同类评测。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先理解语言模型随规模提升的缩放规律,再看人类反馈训练如何让模型更会遵循指令,以及此前对语言模型社交智能的质疑,才能读懂这篇用错误信念任务测心智理论的结果。
- 1
- 2
- 3
目标论文中表现最好的模型 GPT-3.5 系列用了这类人类反馈训练,需要知道训练方式的差别。
读监督微调、奖励模型与强化学习三步;理解它如何改变模型的行为。
- 4
此前对大语言模型社交智能和心智理论的负面评估,目标论文用新的任务给出不同的结果。
读它用了哪些心智理论任务,以及得出模型缺乏社交智能的依据。
- 5Evaluating Large Language Models in Theory of Mind Tasks
目标论文:用错误信念任务考察 11 个语言模型,发现新的模型能解出相当比例的任务。
读任务设计(错误信念加三种对照和反转版本)、判分规则,以及不同模型的通过率对比。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“This recent paper suggests that recent LLMs may be acquiring theory of mind (or something analogous to it): Evaluating Large Language Models in Theory of Mind Tasks”
“These findings suggest that ToM-like ability (thus far considered to be uniquely human) may have spontaneously emerged as a byproduct of language models' improving language skills.”
“There's more than comprehension. It can do some amount of higher order reasoning: Evaluating Large Language Models in Theory of Mind Tasks”
“I've seen people draw conclusions from applying Theory of Mind tests to LLMs.”
“Take the theory of mind questions that reveal the LLMs do have a basic theory of mind: Evaluating Large Language Models in Theory of Mind Tasks.”
看全部 10 条 →
“Theory of Mind May Have Spontaneously Emerged in Large Language Models - Evaluating Large Language Models in Theory of Mind Tasks”
“Another poster shared a link to this paper last week for Theory of Mind: Evaluating Large Language Models in Theory of Mind Tasks”
“Have you seen the latest research, where GPT-4 is shown to have "Theory of Mind"-like ability (what people are thinking) at a human 7-year old level?”
“One of the articles linked here [1] reminded me of the great Ranulph Glanville”
- Theory of Mind May Have Spontaneously Emerged in Large Language Models ↗izzygonzalez · 2023-02-09 · 3 分