按年被提到
20222023 · 16 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2203.02155 · cs.CL

Training language models to follow instructions with human feedback

用人类反馈训练语言模型遵循指令
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton 等 20 人 · 2022

InstructGPT:用人类反馈微调,使 1.3B 模型的输出比 175B GPT-3 更受偏好

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 19 人首次 2022-10-26 · 最近 2026-06-06发表于 neural information processing systems
19不同的人
13活跃月份
22讨论串
5,471被引(OpenAlex)

把语言模型做大并不会自动让它更符合用户意图,模型可能输出不真实、有毒或无益的内容。作者先收集标注员写的示范数据,用监督学习微调 GPT-3,再收集对模型输出的排序数据,用基于人类反馈的强化学习(RLHF)继续微调,得到 InstructGPT。人类评估中,1.3B 的 InstructGPT 的输出比 175B 的 GPT-3 更受偏好,且真实性提升、有毒输出减少,在公开 NLP 数据集上的性能下降很小;但它仍会犯简单错误。

HN 为什么提它

评论者把它作为讲解 RLHF 与 ChatGPT 指令微调方法的原始出处,同时也引用其中的对齐税,以及人类偏好胜率可能只反映风格而非能力的批评。

奠基之作难度 3 / 5约 70 分钟经得起时间
读之前最好知道
1GPT-3 与语言模型预训练
2监督微调(SFT)
3强化学习与 PPO
4奖励模型(reward model)与人类偏好标注
5模型对齐(alignment)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐nl · 2023-01

“Read Training language models to follow instructions with human feedback if you want all the details (although the HF explainer is easier to follow).”

推荐读它了解全部细节,不过 HF 的解释文章更易懂
Announcement: AI generated answers are officially banned here原评论 ↗
推荐hansvm · 2023-06

“OpenAI's InstructGPT paper [0] is decent and references enough other material to get started.”

推荐 InstructGPT 论文入门 RLHF,写得不错且引用了足够多资料
How RLHF Works原评论 ↗
批评superkuh · 2023-07

“Instruction fine tuning definitely increases the win rate in human judged comparisons but that just means”

批评指令微调只是提高人工评测胜率,这多半只反映风格偏好
GPT-4 is getting worse over time, not better原评论 ↗
引用espadrine · 2022-10

“papers show that various finetuning techniques achieve significant gains”

引用它说明各种微调技术能带来显著提升
GPT-3 can create both sides of an Interactive Fiction transcript原评论 ↗
引用imranq · 2022-12

“The diagram on the chatGPT blogpost is literally the same diagram in the March 2022 paper: Training language models to follow instructions with human feedback”

指出 ChatGPT 博客里的图与这篇 2022 年 3 月论文的图完全相同
A New Chat Bot Is a ‘Code Red’ for Google’s Search Business原评论 ↗
引用mattalex · 2023-02

“In fact, if you look at the original paper Training language models to follow instructions with human feedback Figure 1, you can see that even small models already significantly beat the current SOTA.”

指出看原论文 Figure 1,小模型也已大幅超过当时 SOTA
Open Assistant – project meant to give everyone access to a great chat based LLM原评论 ↗
看全部 20 条 →
引用yayr · 2023-02

“also an approach to solve the alignment problem.”

把它当作解决对齐问题的一种方法
I made ChatGPT and Bing AI have a conversation原评论 ↗
引用nl · 2023-03

“InstructGPT[2] (which OpenAI points at as most relevant ChatGPT publication) doesn't report MMLU performance.”

指出 OpenAI 称最相关的 ChatGPT 论文 InstructGPT 没有报告 MMLU 成绩
Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model原评论 ↗
引用nl · 2023-03

“We train three model sizes (1.3B, 6B, and 175B parameters)[1]”

引用它训练了 1.3B、6B、175B 三种规模的模型
GPT-4 is coming next week – and it will be multimodal, says Microsoft Germany原评论 ↗
引用jefftk · 2023-03

“as was most of the later development [2][3][4][5].”

指出 InstructGPT 与后来的大部分发展一脉相承
Microsoft lays off an ethical AI team as it doubles down on OpenAI原评论 ↗
引用make3 · 2023-03

“ChatGPT is further trained with reinforcement from human feedback to make them more tool-like”

引用它说明 ChatGPT 经人类反馈强化学习后更像工具
Show HN: ChatLLaMA – A ChatGPT style chatbot for Facebook's LLaMA原评论 ↗
引用vellum · 2023-05

“By default, when we train a PPO model on our API distribution, it suffers from an “alignment tax”, as its performance on several public NLP datasets decreases.”

引用其“对齐税”说法:PPO 训练后模型在一些公开 NLP 数据集上表现下降
Has ChatGPT Been Neutered?原评论 ↗
引用joshka · 2023-05

“It had me wondering whether this instruction and the resulting training still had a tendency to train these models too far in the wrong direction, to be agreeable and wrong rather than right.”

读了标注员指令后担心训练是否让模型变得迎合而非正确
Lawyer cites fake cases invented by ChatGPT, judge is not amused原评论 ↗
引用vellum · 2023-05

“The InstructGPT models perform worse on a lot of benchmarks.”

指出 InstructGPT 模型在许多基准上表现更差
Chain-of-Thought Hub: Measuring LLMs' Reasoning Performance原评论 ↗
引用senko · 2023-06

“(academic paper, so much denser than the above two resources)”

把它列为学术论文,比前面两个资源密得多
How RLHF Works原评论 ↗
引用simbolit · 2023-10

“I think this premiered with the InstructGPT paper: Training language models to follow instructions with human feedback”

指出这种指令调优大概最早出现在 InstructGPT 论文里
All of ChatGPT's System Prompts原评论 ↗
引用throwaway4aday · 2024-06

“The resulting InstructGPT models are much better at following instructions than GPT-3.”

引用其结果:InstructGPT 比 GPT-3 更会遵循指令
LLMs aren't "trained on the internet" anymore原评论 ↗
引用bbor · 2024-06

“This is the most egregious I've found -- the RLHF paper: see Training language models to follow instructions with human feedback”

举 RLHF 论文为最严重的例子:许多作者已离开 OpenAI 和 Anthropic
Safe Superintelligence Inc.原评论 ↗
引用godelski · 2025-08

“The Instruct-GPT paper was just the first time someone integrated RLHF into a LLM (but not a LM).”

指出 InstructGPT 论文只是首次把 RLHF 整合进 LLM(而非普通 LM)
Curious about the training data of OpenAI's new GPT-OSS models? I was too原评论 ↗
引用CompleteSkeptic · 2026-06

“later got included in the paper[2]”

指出某个轶事式的查询示例后来被收入这篇论文
What was your "Oh Shit" moment with GenAI?原评论 ↗