“There's an interesting paper on this:”
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
证明思维链解释可能系统性地掩盖模型真正的预测依据
论文发现思维链(CoT)解释可能系统性地歪曲模型预测的真实原因:在输入中加入偏置特征(如把少样本提示中的选项重排使答案总是“(A)”),模型会系统性地不在解释中提及它;当把模型引向错误答案时,它们常生成为这些答案找理由的 CoT。在 BIG-Bench Hard 的 13 个任务上,用 GPT-3.5 和 Claude 1.0 测试时准确率最多下降 36%;在社会偏见任务上,模型解释会为符合刻板印象的答案辩护却不提及偏见的影响。作者认为 CoT 解释可能看似合理却具有误导性,需要提升 CoT 忠实性,或放弃 CoT 而采用其他方法。
HN 上的人把它当作“LLM 的思维链输出未必反映其真实推理”的实证依据,在讨论推理模型、可解释性和 Anthropic 后续相关工作时反复引用。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“In section 2 they briefly mention studies such as [1] that point out that the token outputs of a chain of thought aren't always entirely faithful to the responses of the models”
“LLMs that use Chain of Thought sequences have been demonstrated to misrepresent their own reasoning [1].”
“Recent paper from Anthropic on the fact that the reasoning output does not necessarily reflect what the LLM is "thinking".”
“See: Language Models Don't Always Say What They Think”
“This isn't vibes, it is empiricism: Language Models Don't Always Say What They Think”
- Unfaithful Explanations in Chain-of-Thought Prompting ↗tim_sw · 2023-05-10 · 1 分