“Hattie Zhou has a good paper on that as well.”
Teaching Algorithmic Reasoning via In-context Learning
算法提示:在提示里写清算法步骤,让 LLM 做对多位算术
识别出教会 LLM 算法推理的四个关键阶段:把算法表述为技能、同时教多个技能(技能累积)、教如何组合技能(技能组合)、以及教如何把技能当作工具使用,并展示可通过上下文学习(即算法提示)实现。在算术和定量推理任务上显著优于现有提示技术;长奇偶、加法、乘法、减法的错误率分别降低约 10、9、5、2 倍。
在 2023 年关于 LLM 能否推理、能否做算术的争论中,评论者反复引用它作证据:把加法描述成算法后,GPT 系模型的加法准确率大幅提升,说明靠提示就能激发其推理能力。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“If you simply describe addition as an algorithm and ask language models to perform said algorithm on 2 numbers, addition arithmetic accuracy even on very large numbers shoots up to 98.5%”
“Yet if you define the algorithmic steps to perform addition on 2 numbers, accuracy on addition arithmetic shoots up to 98% even on very large numbers. Teaching Algorithmic Reasoning via In-context Learning”
“That means that even with the giant model, you need to stuff even the most basic knowledge for dealing with problems of that class into the prompt space to get it to work”
“You can increase the accuracy of arithmetic by simply describing arithmetic (addition, subtraction etc) as an algorithm to be performed on the numbers.”
“Describe arithmetic as an algorithm to be performed on two numbers.”
看全部 20 条 →
“Oh boy, if you want to fling papers/research, i can do that just fine.”
“You can get GPT-3 (yes 3) to have 98.5% accuracy on addition arithmetic (even very large numbers) by..simply describing the algorithm of addition to performed on 2 numbers.”
“However, simply by describing arithmetic as an algorithm to be performed, you can supercharge accuracy to basically 100% for up to 13 digit addition and >90% after.”
“100% accuracy on up to 13 digit addition can be taught to 3.5 as is.”
“Not too shocking for me after this paper.”
“Your approach reminds me of Hattie's work: Teaching Algorithmic Reasoning via In-context Learning”
“Does it though? When allowing LLMs to use their outputs as a form of state they can very much succeed up to 14 digits”
“There's many papers at this point demonstrating pretty strong reasoning.”
“you can basically take your pick of reasonong type - Algorithmic, Casual, Inference, Analogical and read a paper showing strong performance.”
“You are wrong. and there's many papers to show otherwise.”
“Teaching Algorithmic Reasoning via In-context Learning - new prompting method introduced to improve algorithmic calculations. dataset concerns make no sense.”
“You can greatly increase GPT's arithmetic capabilities tackling it like a problem to solve "on paper" in context.”
“alright with algorithmic prompts - Teaching Algorithmic Reasoning via In-context Learning”
“You can juice arithmetic performance as is with algorithmic instructions.”