“Figure 2 is not at the logit level, it's at the whole completion level (entire names classified by nationality).”
Creativity Has Left the Chat: The Price of Debiasing Language Models
RLHF对齐后的模型输出变单一,创造力下降
论文以 Llama-2 系列做了三组实验,研究 RLHF 对语言模型创造力(句法和语义多样性)的意外影响。结果显示对齐后的模型 token 预测熵更低,在嵌入空间中形成彼此分离的簇,并趋向“吸引子状态”,说明输出多样性有限。作者提醒依赖 LLM 做文案等创意工作的人要权衡一致性与创造力,并讨论了用提示工程发挥基础模型创造力的重要性。
HN 评论者反复拿它说明 RLHF 对齐会造成模式坍塌、降低模型创造力,并讨论原因是否在于 PPO 丢弃了许多有效的 token 轨迹。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“See the shocking results of this paper for instance: Creativity Has Left the Chat”
“Agentic behavior is instilled in RLHF”
“This paper, for example, attributes most of it to PPO discarding a lot of valid token trajectories during RLHF, and not some inevitable emergent effect.”
“Research like [1] suggests algorithms like PPO as one of the possible culprits in the lack of variety, as they can filter out entire token trajectories.”
“This was discussed in my paper last year: Creativity Has Left the Chat”
看全部 8 条 →
“there's evidence that alignment also significantly reduces model creativity: Creativity Has Left the Chat”
“Even this paper shows how model creativity significantly drops and the models experience mode collapse”
- Creativity Has Left the Chat: The Price of Debiasing Language Models ↗behnamoh · 2024-06-11 · 6 分
- Creativity Has Left the Chat: The Price of Debiasing Language Models ↗cubefox · 2024-06-13 · 3 分
- Creativity has left the chat: The price of debiasing language models ↗hardmaru · 2024-06-17 · 174 分