“To me, this looks like the sort of careful, specific and objective work that will lead to us a better understanding of our concepts of the mental.”
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
符号模板基准显示 LLM 数学推理很脆弱
针对 GSM8K 分数提高后是否真有推理进步这一疑问,提出由符号模板生成多样化题目的 GSM-Symbolic 基准,并对多个开源与闭源模型做大规模研究。结果显示,同一道题的不同实例化会让模型表现出明显方差,仅改变数值就会使所有模型表现下降,题目子句增多时性能显著恶化,加入一条看似相关却无助于解题的子句会让所有最先进模型最高掉 65%。作者据此推测,当前 LLM 并非进行真正的逻辑推理,而是复现训练数据中的推理步骤。
评论者把它当作 LLM 并不真正推理的证据,并推荐这种变量化评测优于排行榜;也有人指出它只测了 GPT-4o 等有限模型。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“A better approach is to use variable evaluation like GSM-Symbolic (for evaluating mathematic reasoning): GSM-Symbolic”
“This paper does an excellent job of proving this point with some elegant examples: GSM-Symbolic”
“I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions”
“It also only looked at GPT-4o, ignoring the allegedly more expensive GPT-4.”
“They cannot reason [1].”
看全部 13 条 →
“did you not read GSM-Symbolic”
“An 18% drop in accuracy (figure 8) is not insignificant.”
“A person understanding higher level math, would never (and even less so, consistently) fail a problem like:”
“this work is similar to the GSM symbolic paper (applied to putnam) GSM-Symbolic”
“That tracks with this [0] preprint also released by Apple a few months ago.”
“LLMs are not capable of symbolic reasoning[0] and if you understand how they work internally, you will realize they do no reasoning whatsoever.”
“it seems even there they are mostly generalising from vast amounts of training data, rather than actually reasoning: GSM-Symbolic”
- Understanding the Limitations of Mathematical Reasoning in LLMs ↗hnhn34 · 2024-10-11 · 282 分