按年被提到
20232023 · 1 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2309.00267 · cs.CL

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-09-04 · 最近 2024-05-24
2不同的人
2活跃月份
2讨论串
319被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论astrange · 2024-05

“The previous model does. https://arxiv.org/abs/2309.00267 That's during fine-tuning though. I know I read one about applying it during pretraining, maybe this one? https://arxiv.org/abs/2302.08582”

Google says no "African countries beginning with K" but Kenya has a "K sound"原评论 ↗
被提交为 story · 1 次