按年被提到
20242024 · 1 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2402.04792 · cs.AI

Direct Language Model Alignment from Online AI Feedback

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, Mathieu Blondel · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-02-08 · 最近 2025-03-07
2不同的人
2活跃月份
2讨论串
52被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model. However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline. Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy. In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as annotator: on each training iteration, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback. Despite its simplicity, we demonstrate via human evaluation in several tasks that OAIF outperforms both offline DAP and RLHF methods. We further show that the feedback leveraged in OAIF is easily controllable, via instruction prompts to the LLM annotator.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论kiratp · 2025-03

“Unless I’m missing something this isn’t online RL. They are collecting outputs in one pass and then doing a separate offline GRPO training run on those. The results of this paper would indicate doing what they did, but online could return better results https://arxiv.org/abs/2402.04792”

Using GRPO to Beat o1, o3-mini and R1 at “Temporal Clue”原评论 ↗
被提交为 story · 1 次