“Oh boy, if you want to fling papers/research, i can do that just fine.”
Emergent Analogical Reasoning in Large Language Models
GPT-3 在多种类比任务上的表现接近甚至超过人类。
把人类被试与 GPT-3(text-davinci-003)放在一系列类比任务上直接对比,其中包括基于瑞文标准推理测验规则结构的非视觉矩阵推理任务。GPT-3 表现出很强的抽象模式归纳能力,在多数设置下与人类持平甚至更高;对 GPT-4 的初步测试表现更好。作者据此认为,GPT-3 这类大语言模型已涌现出零样本解决多种类比问题的能力。
在 2023 年关于 LLM 能否做抽象推理的争论中,评论者反复引它作为“LLM 在类比推理上不输人类、接近顶尖水平”的证据,并强调其测试集是新做的、不易被污染。
学习路径
读这篇之前,按顺序读书架上的 6 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径建立 GPT-3 类模型、涌现能力和指令微调的背景,再引入认知科学里对类人推理和神经符号方法的讨论,用来理解 GPT-3 类比推理实验的意义。
- 1
目标论文测试的 GPT-3 系列模型,零样本和少样本能力来自这里。
读few-shot 和 zero-shot 的任务设置,以及规模与表现的关系。
- 2
目标论文使用的 text-davinci-003 属于经人类反馈微调的这一类模型。
读人类反馈微调的流程和它对模型行为的影响。
- 3
目标论文讨论类比推理是否随规模涌现,这是涌现能力的核心概念。
读涌现能力的定义和随规模出现突变的例子;理解为什么这个概念有争议。
- 4
认知科学视角:人类智能的特点,包括类比、组合性,是判断模型是否类人的参照。
读关于因果模型、直觉理论和组合性的论述。
- 5
区分语言能力与思维能力,帮助评价语言模型在推理任务上的表现。
读形式语言能力与功能语言能力的区分。
- 6
目标论文用到矩阵推理类任务,这篇是用神经符号方法解瑞文矩阵的对照。
读瑞文渐进矩阵任务的形式,以及向量符号表示的做法。
- 7Emergent Analogical Reasoning in Large Language Models
目标论文:比较 GPT-3 和人类在多种类比任务上的表现。
读各类类比任务的设计、人与模型的对比结果和作者的解读。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Yeah. Emergent Analogical Reasoning in Large Language Models”
“For the text, LLMs don't really have any problem with analogical reasoning Emergent Analogical Reasoning in Large Language Models”
“LLMs don't really have any problem with abstract analogical reasoning”
“For abstract analogical reasoning, LLMs don't have a problem with that. Emergent Analogical Reasoning in Large Language Models”
“You're right that LLMs can solve abstract analogical reasoning problems Emergent Analogical Reasoning in Large Language Models”
看全部 15 条 →
“In some cases (nlp, analogical reasoning - Emergent Analogical Reasoning in Large Language Models, and some others it's top %)”
“For some like abstract analogical reasoning, LLMs are top % performers.”
“you can basically take your pick of reasonong type - Algorithmic, Casual, Inference, Analogical and read a paper showing strong performance.”
“You are wrong. and there's many papers to show otherwise.”
“Emergent Analogical Reasoning in Large Language Models - new evaluation set introduced with the paper. modelled after tests that previously only had visual equivalents. contamination literally impossible”
“It doesn't even hold with your example because GPT-4 is pretty good at Math, nowhere near”
“LLMs are not bad at abstraction in general, just spatially focused abstraction.”
“Well you are wrong”
“they’re about as good as humans on a text version of Raven’s matrices, another IQ test.”
- Emergent Analogical Reasoning in Large Language Models ↗mfiguiere · 2023-03-27 · 1 分
- GPT-3 outperforms humans on Raven's matrices, shows strong analogical reasoning ↗og_kalu · 2023-04-04 · 3 分