“Have you had a chance to look at the Direct Preference Optimization [1] pre-print?”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
用简单分类损失替代 RLHF 的偏好对齐方法
针对 RLHF 流程复杂且不稳定(先拟合奖励模型,再用强化学习最大化奖励并限制偏离原模型)的问题,提出对 RLHF 奖励模型的新参数化,使最优策略能以闭式解给出,从而只需一个简单的分类损失就能解决标准 RLHF 问题。由此得到的直接偏好优化(DPO)稳定、性能好、计算轻量,微调时不需要从模型采样或大量调超参。实验显示,DPO 在对齐人类偏好上与现有方法持平或更好,在控制生成情感上超过基于 PPO 的 RLHF,在摘要和单轮对话上匹配或提高回答质量,且实现和训练都简单得多。
评论者把它当作不用强化学习的偏好对齐方法的代表,常用来说明绕开 RLHF 的一条更简单路径,并以此推测各家模型的微调方式。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“There was an interesting paper recently suggesting that you can use a preference learning objective directly”
“See the abstract of the DPO paper Direct Preference Optimization, and Figure 1 in the paper”
“using the techniques described in the Direct Preference Optimization”
“You should note that Claude Design is most likely a DPO play: Direct Preference Optimization”
“Attack the RLHF problem with a simple classification loss: Direct Preference Optimization”
- RLFH Alternative Direct Preference Optimization ↗nl · 2023-05-30 · 2 分
- Direct Preference Optimization: Your Language Model Is a Reward Model ↗tim_sw · 2023-05-31 · 2 分
- Direct Preference Optimization: Your Language Model Is a Reward Model ↗ntonozzi · 2023-06-05 · 1 分
- Direct Preference Optimization: Your Language Model Is Secretly A Reward Model ↗optimalsolver · 2023-09-05 · 1 分
- Direct Preference Optimization: Your Language Model Is a Reward Model ↗tosh · 2024-01-11 · 3 分
- Direct Preference Optimization: Your Language Model Is a Reward Model ↗rahimnathwani · 2024-01-12 · 1 分