按年被提到
20222023 · 23 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2211.09066 · cs.LG

Teaching Algorithmic Reasoning via In-context Learning

通过上下文学习教会算法推理
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, Hanie Sedghi · 2022

算法提示:在提示里写清算法步骤,让 LLM 做对多位算术

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 6 人首次 2022-12-07 · 最近 2023-12-15
6不同的人
9活跃月份
19讨论串
44被引(OpenAlex)

识别出教会 LLM 算法推理的四个关键阶段:把算法表述为技能、同时教多个技能(技能累积)、教如何组合技能(技能组合)、以及教如何把技能当作工具使用,并展示可通过上下文学习(即算法提示)实现。在算术和定量推理任务上显著优于现有提示技术;长奇偶、加法、乘法、减法的错误率分别降低约 10、9、5、2 倍。

HN 为什么提它

在 2023 年关于 LLM 能否推理、能否做算术的争论中,评论者反复引用它作证据:把加法描述成算法后,GPT 系模型的加法准确率大幅提升,说明靠提示就能激发其推理能力。

方法难度 3 / 5约 35 分钟有时效性
读之前最好知道
1大语言模型与上下文学习(in-context learning)
2思维链(chain-of-thought)提示
3算术与奇偶性任务
4提示工程

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐chaxor · 2023-05

“Hattie Zhou has a good paper on that as well.”

推荐 Hattie Zhou 的这篇好论文,讲 Transformer 的算法推理
It's not just statistics: GPT-4 does reason原评论 ↗
引用og_kalu · 2023-03

“If you simply describe addition as an algorithm and ask language models to perform said algorithm on 2 numbers, addition arithmetic accuracy even on very large numbers shoots up to 98.5%”

引它说明把加法描述成算法后,语言模型的大数加法准确率升到 98.5%
Bing AI can play chess原评论 ↗
引用og_kalu · 2023-03

“Yet if you define the algorithmic steps to perform addition on 2 numbers, accuracy on addition arithmetic shoots up to 98% even on very large numbers. Teaching Algorithmic Reasoning via In-context Learning”

引它说明写明加法的算法步骤后,大数加法准确率升到 98%
OK, it’s time to freak out about AI原评论 ↗
引用dragonwriter · 2023-03

“That means that even with the giant model, you need to stuff even the most basic knowledge for dealing with problems of that class into the prompt space to get it to work”

引它说明即使是大模型,也得把这类问题的基本知识塞进提示里
OK, it’s time to freak out about AI原评论 ↗
引用og_kalu · 2023-03

“You can increase the accuracy of arithmetic by simply describing arithmetic (addition, subtraction etc) as an algorithm to be performed on the numbers.”

引它说明把算术描述成算法就能提高 GPT-3 的准确率到 98.5%
And yet It Understands原评论 ↗
引用og_kalu · 2023-03

“Describe arithmetic as an algorithm to be performed on two numbers.”

引它说明把算术描述成算法就能准确做大数加法
And yet It Understands原评论 ↗
看全部 20 条 →
引用og_kalu · 2023-03

“Oh boy, if you want to fling papers/research, i can do that just fine.”

拿它作为反驳对方的证据,说明可以通过上下文学习做算法推理
引用og_kalu · 2023-03

“You can get GPT-3 (yes 3) to have 98.5% accuracy on addition arithmetic (even very large numbers) by..simply describing the algorithm of addition to performed on 2 numbers.”

引它说明描述加法算法就能让 GPT-3 加法准确率达 98.5%
Do large language models need sensory grounding for meaning and understanding?原评论 ↗
引用og_kalu · 2023-04

“However, simply by describing arithmetic as an algorithm to be performed, you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after.”

引它说明把算术描述成算法能让 13 位以内加法接近 100%
Teaching ChatGPT to Speak My Son’s Invented Language原评论 ↗
引用og_kalu · 2023-04

“100% accuracy on up to 13 digit addition can be taught to 3.5 as is.”

引它说明 GPT-3.5 能被教会 13 位加法,准确率 100%
Emergent abilities of large language models原评论 ↗
引用og_kalu · 2023-04

“Not too shocking for me after this paper.”

表示读过这篇论文后对此并不意外,上下文学习无需改权重就能教会算术
Teaching Large Language Models to Self-Debug原评论 ↗
引用shawntan · 2023-04

“Your approach reminds me of Hattie's work: Teaching Algorithmic Reasoning via In-context Learning”

觉得对方的做法让人想起 Hattie 这篇关于上下文教算法的工作
The New XOR Problem原评论 ↗
引用PartiallyTyped · 2023-04

“Does it though? When allowing LLMs to use their outputs as a form of state they can very much succeed up to 14 digits”

反驳说让 LLM 把输出当作状态,多位数运算最多可做到 14 位
Study: ChatGPT Outperforms Physicians in Quality, Empathetic Answers to Patients原评论 ↗
引用og_kalu · 2023-05

“There's many papers at this point demonstrating pretty strong reasoning.”

引它说明已有许多论文显示 LLM 能靠提示做算法推理
LLMs are not greater than the sum of their parts: Stanford researchers原评论 ↗
引用og_kalu · 2023-05

“you can basically take your pick of reasonong type - Algorithmic, Casual, Inference, Analogical and read a paper showing strong performance.”

拿它说明算法推理等各类推理都有论文显示 LLM 表现很强
Just Calm Down About GPT-4 Already原评论 ↗
引用og_kalu · 2023-05

“You are wrong. and there's many papers to show otherwise.”

反驳对方,称有很多论文证明 LLM 能推理,此文即其一
Governance of Superintelligence原评论 ↗
引用og_kalu · 2023-05

“Teaching Algorithmic Reasoning via In-context Learning - new prompting method introduced to improve algorithmic calculations. dataset concerns make no sense.”

介绍它提出的新提示方法,能提升算法计算,并认为数据集担忧站不住
Governance of Superintelligence原评论 ↗
引用og_kalu · 2023-06

“You can greatly increase GPT's arithmetic capabilities tackling it like a problem to solve "on paper" in context.”

拿它说明在上下文里像“纸上演算”一样教步骤,能大幅提升 GPT 算术
引用og_kalu · 2023-07

“alright with algorithmic prompts - Teaching Algorithmic Reasoning via In-context Learning”

用它说明加上算法式提示后,LLM 算术还不错
OpenAI: Introducing SuperAlignment原评论 ↗
引用og_kalu · 2023-07

“You can juice arithmetic performance as is with algorithmic instructions.”

拿它说明用算法式指令就能直接提升 LLM 的算术表现
AI Safety and the Age of Dislightenment原评论 ↗