批评curious_cat_163 · 2025-06
“On one of these toy problems, the Tower of Hanoi, the models structurally cannot output enough tokens to solve the problem”
批评汉诺塔失败是 token 数量受限所致,不能说明推理能力不足
用可控谜题测试推理模型,复杂度过高时准确率彻底崩溃
论文用可精确调节复杂度的谜题环境评估大型推理模型(LRM),不仅看最终答案,也分析推理轨迹。实验显示 LRM 在超过某一复杂度后准确率完全崩溃,推理努力在问题变难时先升后降,尽管仍有 token 预算。对比同等推理算力下的普通 LLM,出现三种区间:低复杂度时普通模型更好,中等复杂度时 LRM 占优,高复杂度时两者都崩溃;LRM 在精确计算上也有局限,不会使用显式算法,在不同规模上推理不一致。
HN 评论者引它支持“推理模型只是更高层次的模式匹配、到复杂度极限就失效”的观点,也有人批评如汉诺塔这类题的失败其实源于输出 token 上限,而不是推理能力。
“On one of these toy problems, the Tower of Hanoi, the models structurally cannot output enough tokens to solve the problem”
“just pattern matching at a higher level, and eventually runs out of steam.[0]”
“in 2025 this paper showed that "thought traces" in the models of the time could sometimes be inaccurate or misleading.”
“AI models and in particular LLMs are not capable of logic reasoning. See for example this paper:”