“why spend the enormous effort running PPO to brute force the classifier to obtain desired text or image output”
Fine-Tuning Language Models from Human Preferences
早期 RLHF 工作:用人类偏好比较训练奖励模型来微调 GPT-2
论文把奖励学习应用到四个自然语言任务:续写带积极情感或物理描述性的文本,以及 TL;DR 和 CNN/Daily Mail 摘要。风格续写只用 5,000 次人工比较就取得不错效果;摘要任务中用 60,000 次比较训练的模型会照搬输入句子但跳过无关开头,ROUGE 合理、人工标注者评价很高,但可能利用了标注者依赖简单启发式的特点。
HN 评论者把它当作 RLHF 的早期源头来追溯引用:用人类偏好比较训练奖励模型,再用 PPO 微调预训练的 GPT-2。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径先建立 Transformer 语言模型与预训练微调的基础,再补上 PPO 和摘要任务的背景,然后读用人类偏好微调语言模型的早期 RLHF 工作。
- 1
- 2
预训练再微调的范式,论文是在此基础上用奖励学习去微调。
读预训练语言模型迁移到下游任务的做法。
- 3
- 4
- 5Fine-Tuning Language Models from Human Preferences
目标论文:用人类偏好比较训练奖励模型,再用强化学习微调 GPT-2。
读奖励模型的训练、KL 惩罚、情感与摘要任务的结果。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“as was most of the later development [2][3][4][5].”
“if you trace the citations back to [1], you can see that the policy is initialized with a pretrained language model (in that case, GPT-2).”
“2019 coauthor on RLHF Fine-Tuning Language Models from Human Preferences)”
“is that these models are capable of self-correcting behaviour — we just don't do that live because that makes it much harder to research and to use”
“Fine-Tuning Language Models from Human Preferences fig 1, and also section 4.4 for what can go wrong”