按年被提到
20242025 · 1 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2402.08115 · cs.AI

On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

Kaya Stechly, Karthik Valmeekam, Subbarao Kambhampati · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-04-30 · 最近 2026-08-13
2不同的人
2活跃月份
2讨论串
35被引(OpenAlex)
arXiv 摘要(候选层不写导读)

There has been considerable divergence of opinion on the reasoning abilities of Large Language Models (LLMs). While the initial optimism that reasoning might emerge automatically with scale has been tempered thanks to a slew of counterexamples--ranging from multiplication to simple planning--there persists a wide spread belief that LLMs can self-critique and improve their own solutions in an iterative fashion. This belief seemingly rests on the assumption that verification of correctness should be easier than generation--a rather classical argument from computational complexity--which should be irrelevant to LLMs to the extent that what they are doing is approximate retrieval. In this paper, we set out to systematically investigate the effectiveness of iterative prompting in the context of reasoning and planning. We present a principled empirical study of the performance of GPT-4 in three domains: Game of 24, Graph Coloring, and STRIPS planning. We experiment both with the model critiquing its own answers and with an external correct reasoner verifying proposed solutions. In each case, we analyze whether the content of criticisms actually affects bottom line performance, and whether we can ablate elements of the augmented system without losing performance. We observe significant performance collapse with self-critique and significant performance gains with sound external verification. We also note that merely re-prompting with a sound verifier maintains most of the benefits of more involved setups.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论dudeinhawaii · 2025-04

“I see a lot of threads pitting models against each other (or whole swarms of them) in the hope that "wisdom of crowds" will magically appear. After a stack of experiments of my own—and after watching the recent ASU/Microsoft-Research work [1].. I've landed on a simpler takeaway: An LLM is a terrible verifier of another LLM. Subbarao Kambhampati's "(How) …”

Chain of Recursive Thoughts: Make AI think harder by making it argue with itself原评论 ↗
评论YeGoblynQueenne · 2026-08

“Thanks, but I can't see that in the article by Anthropic. AFAICT the "650 ideas" that were wrong were all generated independently of each other. I don't agree that LLMs can evaluate an experiment. There's nothing in LLM training that makes them capable of telling what is e.g. a correct hypothesis from an incorrect one. I know that is a common claim parti…”

Tim Gowers: What sort of maths are LLMs good at?原评论 ↗