按年被提到
20252025 · 3 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2505.05410 · cs.CL

Reasoning Models Don't Always Say What They Think

Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman 等 15 人 · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2025-05-10 · 最近 2025-06-11
3不同的人
2活跃月份
3讨论串
123被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Chain-of-thought (CoT) offers a potential boon for AI safety as it allows monitoring a model's CoT to try to understand its intentions and reasoning processes. However, the effectiveness of such monitoring hinges on CoTs faithfully representing models' actual reasoning processes. We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out. They also suggest that in settings like ours where CoT reasoning is not necessary, test-time monitoring of CoTs is unlikely to reliably catch rare and catastrophic unexpected behaviors.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论cluckindan · 2025-06

“They don’t ”think”. https://arxiv.org/abs/2503.09211 They don’t ”reason”. https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf They don’t even always output their internal state accurately. https://arxiv.org/abs/2505.05410”

Magistral — the first reasoning model by Mistral AI原评论 ↗
被提交为 story · 2 次