“Im dont reading LLM papers. Based on what we know of the MS deal (they can have all of open AI's work till AGI is achieved). And with the departure of any one who thought safety was a need something became very clear to me. They thought that they were going to hit some magic threshold of data where it would snap over to some sort of sentience, or sapien…”
LLMs achieve adult human performance on higher-order theory of mind tasks
手写测试集比较五个 LLM 与成年人的高阶心智理论表现
论文用自编的手写测试集 Multi-Order Theory of Mind Q&A,将五个 LLM 与新收集的成年人基准做比较,考察递归推理多个心理状态的高阶心智理论能力。结果显示 GPT-4 和 Flan-PaLM 总体达到或接近成年人水平,GPT-4 在六阶推断上超过成年人;作者认为模型规模与微调共同作用于这种能力,并指出这对面向用户的 LLM 应用有重要含义。
mentions 中没有留存评论摘录,只知道有 3 位用户在 3 个不同月份提到它,具体讨论角度无法确认。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮你建立语言模型心智理论评测的背景:先看对 LLM 社交智能的质疑与测试,再读目标论文的高阶心智理论测试。
- 1
较早质疑大模型的社交智能与心智理论,是后续评测的对照背景。
读它如何测试模型的心智推理,以及为何结论偏悲观。
- 2
用错误信念任务系统测多个 LLM,是目标论文直接承接的评测。
读错误信念任务的设计和各模型的成绩,注意随规模增长的趋势。
- 3
展示稍作改动就让 LLM 在心智理论任务上失败,提醒测试设计要稳健。
读用了哪些微小改动,以及它对模型是否真有心智理论的论证。
- 4
GPT-4 早期能力测试里含心智理论内容,是目标论文所用模型的背景。
读只看其中关于心智理论的部分,其余可略读。
- 5LLMs achieve adult human performance on higher-order theory of mind tasks
目标论文:手写高阶心智理论测试集,对比 LLM 与成年人。
读多阶心智理论题的构造、人类基线,以及 GPT-4 等模型与成人的比较。
HN 书评
没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)“These models are static snapshots of an arbitrary level of intelligence. Are there any small LLMs which are not static and can keep updating in real time with external or internal feedback?”
“That talk[1] (also paper[2]) covering some of this by the Microsoft team involved with the unfettered GPT-4 model certainly shifted my views last year at a time when until then it was more common to see downplaying about LLMs based on prior gen models. Recently saw an AI safety specialist who also become more concerned about the rapidity of progress cite …”
- LLMs achieve adult human performance on higher-order theory of mind tasks ↗amichail · 2024-05-31 · 1 分
- LLMs achieve adult human performance on higher-order theory of mind tasks ↗tosh · 2024-06-03 · 2 分
- Large language models have developed a higher-order theory of mind ↗some-unique-x · 2024-07-02 · 17 分