按年被提到
20242024 · 8 次2026
AI 安全、伦理与社会影响›大模型与人类行为·arXiv:2406.05587 · cs.CL

Creativity Has Left the Chat: The Price of Debiasing Language Models

创造力已离开聊天:语言模型去偏的代价
Behnam Mohammadi · 2024

RLHF对齐后的模型输出变单一,创造力下降

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 5 人首次 2024-06-11 · 最近 2025-12-01
5不同的人
7活跃月份
10讨论串
22被引(OpenAlex)

论文以 Llama-2 系列做了三组实验,研究 RLHF 对语言模型创造力(句法和语义多样性)的意外影响。结果显示对齐后的模型 token 预测熵更低,在嵌入空间中形成彼此分离的簇,并趋向“吸引子状态”,说明输出多样性有限。作者提醒依赖 LLM 做文案等创意工作的人要权衡一致性与创造力,并讨论了用提示工程发挥基础模型创造力的重要性。

HN 为什么提它

HN 评论者反复拿它说明 RLHF 对齐会造成模式坍塌、降低模型创造力,并讨论原因是否在于 PPO 丢弃了许多有效的 token 轨迹。

实证研究难度 2 / 5约 25 分钟经得起时间
读之前最好知道
1RLHF(基于人类反馈的强化学习)
2token 预测熵与采样
3文本嵌入与聚类

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
引用gwern · 2024-06

“Figure 2 is not at the logit level, it's at the whole completion level (entire names classified by nationality).”

指出论文图 2 针对的是整段输出而非 logit 层面
Creativity has left the chat: The price of debiasing language models原评论 ↗
引用behnamoh · 2024-06

“See the shocking results of this paper for instance: Creativity Has Left the Chat”

引它的惊人结果:被审查的模型创造力远低于基座 LLM
Refusal in language models is mediated by a single direction原评论 ↗
引用behnamoh · 2024-07

“Agentic behavior is instilled in RLHF”

借它说明 RLHF 会灌输智能体行为并限制创造性探索
AI Agents That Matter原评论 ↗
引用orbital-decay · 2024-10

“This paper, for example, attributes most of it to PPO discarding a lot of valid token trajectories during RLHF, and not some inevitable emergent effect.”

引它说明多样性下降主因是 RLHF 中 PPO 丢弃有效轨迹,而非涌现效应
Understanding the Limitations of Mathematical Reasoning in LLMs原评论 ↗
引用orbital-decay · 2024-12

“Research like [1] suggests algorithms like PPO as one of the possible culprits in the lack of variety, as they can filter out entire token trajectories.”

引它说明 PPO 等算法会滤掉整条 token 轨迹,可能是多样性不足的原因
Things we learned about LLMs in 2024原评论 ↗
引用behnamoh · 2025-01

“This was discussed in my paper last year: Creativity Has Left the Chat”

指出自己去年的论文已讨论过:RLHF 导致模式坍缩、降低创造力
Entropy of a Large Language Model output原评论 ↗
看全部 8 条 →
引用behnamoh · 2025-05

“there's evidence that alignment also significantly reduces model creativity: Creativity Has Left the Chat”

引它作为证据:对齐会显著降低模型创造力
Alignment is not free: How model upgrades can silence your confidence signals原评论 ↗
引用behnamoh · 2025-12

“Even this paper shows how model creativity significantly drops and the models experience mode collapse”

引它说明去偏后模型创造力大幅下降并出现模式坍缩
Sycophancy is the first LLM "dark pattern"原评论 ↗
被提交为 story · 3 次