“Read Training language models to follow instructions with human feedback if you want all the details (although the HF explainer is easier to follow).”
Training language models to follow instructions with human feedback
InstructGPT:用人类反馈微调,使 1.3B 模型的输出比 175B GPT-3 更受偏好
把语言模型做大并不会自动让它更符合用户意图,模型可能输出不真实、有毒或无益的内容。作者先收集标注员写的示范数据,用监督学习微调 GPT-3,再收集对模型输出的排序数据,用基于人类反馈的强化学习(RLHF)继续微调,得到 InstructGPT。人类评估中,1.3B 的 InstructGPT 的输出比 175B 的 GPT-3 更受偏好,且真实性提升、有毒输出减少,在公开 NLP 数据集上的性能下降很小;但它仍会犯简单错误。
评论者把它作为讲解 RLHF 与 ChatGPT 指令微调方法的原始出处,同时也引用其中的对齐税,以及人类偏好胜率可能只反映风格而非能力的批评。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“OpenAI's InstructGPT paper [0] is decent and references enough other material to get started.”
“Instruction fine tuning definitely increases the win rate in human judged comparisons but that just means”
“papers show that various finetuning techniques achieve significant gains”
“The diagram on the chatGPT blogpost is literally the same diagram in the March 2022 paper: Training language models to follow instructions with human feedback”
“In fact, if you look at the original paper Training language models to follow instructions with human feedback Figure 1, you can see that even small models already significantly beat the current SOTA.”
看全部 20 条 →
“also an approach to solve the alignment problem.”
“InstructGPT[2] (which OpenAI points at as most relevant ChatGPT publication) doesn't report MMLU performance.”
“We train three model sizes (1.3B, 6B, and 175B parameters)[1]”
“as was most of the later development [2][3][4][5].”
“ChatGPT is further trained with reinforcement from human feedback to make them more tool-like”
“By default, when we train a PPO model on our API distribution, it suffers from an “alignment tax”, as its performance on several public NLP datasets decreases.”
“It had me wondering whether this instruction and the resulting training still had a tendency to train these models too far in the wrong direction, to be agreeable and wrong rather than right.”
“The InstructGPT models perform worse on a lot of benchmarks.”
“(academic paper, so much denser than the above two resources)”
“I think this premiered with the InstructGPT paper: Training language models to follow instructions with human feedback”
“The resulting InstructGPT models are much better at following instructions than GPT-3.”
“This is the most egregious I've found -- the RLHF paper: see Training language models to follow instructions with human feedback”
“The Instruct-GPT paper was just the first time someone integrated RLHF into a LLM (but not a LM).”
“later got included in the paper[2]”