按年被提到
20242024 · 3 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2405.04776 · cs.AI

Chain of Thoughtlessness? An Analysis of CoT in Planning

Kaya Stechly, Karthik Valmeekam, Subbarao Kambhampati · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2024-05-09 · 最近 2024-10-19
3不同的人
2活跃月份
3讨论串
29被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Large language model (LLM) performance on reasoning problems typically does not generalize out of distribution. Previous work has claimed that this can be mitigated with chain of thought prompting-a method of demonstrating solution procedures-with the intuition that it is possible to in-context teach an LLM an algorithm for solving the problem. This paper presents a case study of chain of thought on problems from Blocksworld, a classical planning domain, and examines the performance of two state-of-the-art LLMs across two axes: generality of examples given in prompt, and complexity of problems queried with each prompt. While our problems are very simple, we only find meaningful performance improvements from chain of thought prompts when those prompts are exceedingly specific to their problem class, and that those improvements quickly deteriorate as the size n of the query-specified stack grows past the size of stacks shown in the examples. We also create scalable variants of three domains commonly studied in previous CoT papers and demonstrate the existence of similar failure modes. Our results hint that, contrary to previous claims in the literature, CoT's performance improvements do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts. This spotlights drawbacks of chain of thought, especially the sharp tradeoff between possible performance gains and the amount of human labor necessary to generate examples with correct reasoning traces.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论subbarao2z · 2024-05

“[I am the one who wrote that tweet thread]. fwiw, a lot of the questions that are being raised in this thread have been answered in this talk I gave for Google/Deepmind LLM reasoning seminar a few days back: https://www.youtube.com/watch?v=hGXhFa3gzBs (Also, the preprint paper on the CoT study with all the prompts etc is available at https://arxiv.org/abs…”

The "Chain of Thought" Delusions原评论 ↗
评论YeGoblynQueenne · 2024-10

“My thoughts on CoT and the extent to which it "elicits reasoning" align almost perfectly with the criticism of Rao Kambhampati and his students: https://arxiv.org/abs/2405.04776 Their argument is that CoT can only improve performance of LLMs in reasoning tasks when the prompter already knows the answer and can somehow embed it in their prompt. The paper…”

Use Prolog to improve LLM's reasoning原评论 ↗
被提交为 story · 1 次