按年被提到
20252025 · 5 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2506.09250 · cs.AI

Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

A. Lawsen · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 5 人首次 2025-06-14 · 最近 2025-07-05
5不同的人
2活跃月份
5讨论串
14被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit "accuracy collapse" on planning puzzles beyond certain complexity thresholds. We demonstrate that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Our analysis reveals three critical issues: (1) Tower of Hanoi experiments risk exceeding model output token limits, with models explicitly acknowledging these constraints in their outputs; (2) The authors' automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities; (3) Most concerningly, their River Crossing benchmarks include mathematically impossible instances for N > 5 due to insufficient boat capacity, yet models are scored as failures for not solving these unsolvable problems. When we control for these experimental artifacts, by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures. These findings highlight the importance of careful experimental design when evaluating AI reasoning capabilities.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论diimdeep · 2025-06

“Here is another opinion on the original paper - Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity https://arxiv.org/abs/2506.09250”

"The Illusion of Thinking" – Thoughts on This Important Paper原评论 ↗
评论ben_w · 2025-07

“> The Apple Paper: When the Illusion Breaks > Apple's recent research paper, The Illusion of Thinking, tested modern LLMs on classic logic puzzles—Tower of Hanoi, river-crossing tasks, and other structured reasoning challenges. > The results? Performance dropped to zero. > Children can solve these puzzles. AI could not. Ah, no. The AI that Apple test…”

Can AI Know Infinity?原评论 ↗
被提交为 story · 3 次