按年被提到
20242024 · 10 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2410.05229 · cs.LG

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

GSM-Symbolic:理解大语言模型数学推理的局限
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar · 2024

符号模板基准显示 LLM 数学推理很脆弱

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 14 人首次 2024-10-11 · 最近 2026-07-06
14不同的人
6活跃月份
14讨论串
266被引(OpenAlex)

针对 GSM8K 分数提高后是否真有推理进步这一疑问,提出由符号模板生成多样化题目的 GSM-Symbolic 基准,并对多个开源与闭源模型做大规模研究。结果显示,同一道题的不同实例化会让模型表现出明显方差,仅改变数值就会使所有模型表现下降,题目子句增多时性能显著恶化,加入一条看似相关却无助于解题的子句会让所有最先进模型最高掉 65%。作者据此推测,当前 LLM 并非进行真正的逻辑推理,而是复现训练数据中的推理步骤。

HN 为什么提它

评论者把它当作 LLM 并不真正推理的证据,并推荐这种变量化评测优于排行榜;也有人指出它只测了 GPT-4o 等有限模型。

实证研究难度 2 / 5约 40 分钟有时效性
读之前最好知道
1大语言模型基础
2GSM8K 等数学推理基准
3基准污染与评测可靠性
4思维链推理

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐mannykannot · 2024-10

“To me, this looks like the sort of careful, specific and objective work that will lead to us a better understanding of our concepts of the mental.”

称赞这是细致、具体、客观的工作,有助于更好理解心智概念
Language is not essential for the cognitive processes that underlie thought原评论 ↗
推荐carschno · 2024-10

“A better approach is to use variable evaluation like GSM-Symbolic (for evaluating mathematic reasoning): GSM-Symbolic”

推荐用 GSM-Symbolic 这样的变量化评估来衡量数学推理,优于榜单
Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku原评论 ↗
推荐griomnib · 2024-11

“This paper does an excellent job of proving this point with some elegant examples: GSM-Symbolic”

称这篇论文用精巧的例子很好地证明了这一点
OK, I can partly explain the LLM chess weirdness now原评论 ↗
推荐loxias · 2024-12

“I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions”

喜欢 GSM-Symbolic,用符号模板改写很简单的题目作基准
OpenAI O3 breakthrough high score on ARC-AGI-PUB原评论 ↗
批评airstrike · 2024-10

“It also only looked at GPT-4o, ignoring the allegedly more expensive GPT-4.”

ChatGPT's Name Bias and Apple's Findings on AI's Lack of Reasoning原评论 ↗
引用efitz · 2024-10

“They cannot reason [1].”

引它作为 LLM 不能推理的依据
Sabotage evaluations for frontier models原评论 ↗
看全部 13 条 →
引用turtlesdown11 · 2024-10

“did you not read GSM-Symbolic”

反问对方是否没读这篇论文,它认为 LLM 是复制推理而非真推理
OpenAI builds first chip with Broadcom and TSMC, scales back foundry ambition原评论 ↗
引用zamalek · 2024-12

“An 18% drop in accuracy (figure 8) is not insignificant.”

指出图 8 中仅改名字和数字就掉 18% 准确率,并不算小
What happens if we remove 50 percent of Llama?原评论 ↗
引用coldtea · 2024-12

“A person understanding higher level math, would never (and even less so, consistently) fail a problem like:”

借它说明懂高等数学的人不会犯、LLM 却会被无关细节带偏的题
GPT-5 is behind schedule原评论 ↗
引用bwfan123 · 2025-01

“this work is similar to the GSM symbolic paper (applied to putnam) GSM-Symbolic”

指出这项工作与 GSM-Symbolic 相似,只是用到 Putnam 题上
30% drop in O1-preview accuracy when Putnam problems are slightly variated原评论 ↗
引用MontyCarloHall · 2025-06

“That tracks with this [0] preprint also released by Apple a few months ago.”

指出这与苹果几个月前发布的预印本一致
The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]原评论 ↗
引用_petronius · 2025-06

“LLMs are not capable of symbolic reasoning[0] and if you understand how they work internally, you will realize they do no reasoning whatsoever.”

引它主张 LLM 不具备符号推理能力,内部根本没有推理
What happens when people don't understand how AI works原评论 ↗
引用delis-thumbs-7e · 2026-07

“it seems even there they are mostly generalising from vast amounts of training data, rather than actually reasoning: GSM-Symbolic”

引它说明即使在数学上,LLM 也主要靠海量训练数据泛化,而非真正推理
Emily Bender Sets the Record Straight on "Stochastic Parrots"原评论 ↗
被提交为 story · 1 次