引用make3 · 2023-03
“ChatGPT is further trained with reinforcement from human feedback to make them more tool-like”
引用它说明 ChatGPT 经人类反馈强化学习后更像工具
Anthropic 的 RLHF 对齐训练:偏好建模、在线迭代与鲁棒性分析
论文用偏好建模和人类反馈强化学习(RLHF)微调语言模型,使其成为有用且无害的助手。作者发现这种对齐训练几乎能提升所有 NLP 评测的表现,并与 Python 编码、摘要等专项技能的训练兼容。文中还探索了每周用新的人类反馈数据更新偏好模型与 RL 策略的迭代在线训练,并发现 RL 奖励与策略相对初始模型的 KL 散度平方根大致呈线性关系,此外分析了校准、相互竞争的目标和 OOD 检测等。
评论者用它说明 RLHF 让 ChatGPT 这类模型更像工具,并引用“模型超过一定规模后对齐反而提升性能”(对齐税转负)的发现。
“ChatGPT is further trained with reinforcement from human feedback to make them more tool-like”
“starts to become negative at least when trained in RLHF”
“for models past a certain size, alignment increases performance: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.”