“There are some interesting contributions in the paper too.”
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
纯强化学习即可激发大模型推理,无需人工标注的推理轨迹
论文表明,大语言模型的推理能力可以通过纯强化学习(RL)激发出来,无需人工标注的推理轨迹。该 RL 框架促成自我反思、验证和动态调整策略等高级推理模式的涌现,所训练的模型在数学、编程竞赛和 STEM 等可验证任务上优于基于人类示范做传统监督学习的对照模型;这些涌现的推理模式还可被系统地用来增强较小模型的推理能力。
评论者把它当作理解 DeepSeek 如何以较少算力得到强推理模型的起点,常引用其中的 GRPO、「顿悟时刻」、失败尝试(MCTS、PRM)、语言混用以及蒸馏小模型等细节。
学习路径
读这篇之前,按顺序读书架上的 3 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先搞清 R1 用的强化学习算法(PPO 到 GRPO)和它所基于的 DeepSeek-V3 基座,再读 R1 如何用纯 RL 激发推理。
- 1
GRPO 是 PPO 的变体,先要懂 PPO 的策略梯度和裁剪目标。
读clipped surrogate objective 与优势估计(advantage)的概念;跳过 Atari 与机器人实验细节。
- 2
R1 的 RL 算法 GRPO 正是在这篇里提出。
读GRPO 一节:用组内相对奖励代替价值网络(critic)做基线;对比 PPO 的差别。
- 3
DeepSeek-R1 的基座模型 R1-Zero 与 R1 都建立在 V3 之上。
读只看模型架构概览(MoE、MLA)和后训练部分;训练基础设施可略读。
- 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
目标论文:在上述基座和 GRPO 之上,用规则奖励做纯 RL,得到长思维链推理。
读R1-Zero 的奖励设计与训练过程中推理长度增长、顿悟时刻;再看冷启动数据和多阶段流程。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The paper for R1 (1) seems to be pretty clear how they got such good results with so little horsepower (see: 'Group Relative Policy Optimization').”
“Many recent OSS models have great tech reports where you can learn more about these kind of things:”
“Here is the R1 paper: DeepSeek-R1”
“If you check failure section of their paper, they also tried other methods like MCTS and PRM which is what other labs have been obsessing about but couldn't move on from (that includes bigshots).”
“Link [2] to the result on more standard LLM benchmarks. They conveniently placed the results on the first page of the paper.”
看全部 20 条 →
“The R1 paper emphasizes their success with reinforcement learning without requiring any supervised data (unlike RLHF for example).”
“it’s explicitly spelled out in DeepSeek’s R1 paper that they had trouble with improving over DeepSeek-V3”
“and DeepSeek-R1 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps.”
“Zero evidence that the above statement is true, and weak evidence (authors' claims) that it is false. Have you read their papers even?”
“DeepSeek talked about “aha moments” in DeepSeek-R1 where under RL the model learned to backtrack and correct itself, for example.”
“DeepSeek released a distilled version based on a 1.5B Qwen model with the full name DeepSeek-R1-Distill-Qwen-1.5B, see chapter 3.2 on page 14 of their research article [0].”
“DeepSeek-R1 had an RLHF step in their post-training pipeline (section 2.3.4 of their technical report[1]).”
“[0]: [LINK] [1]: DeepSeek-R1 [2]: [LINK] [3]: [LINK]”
“in the DeepSeek R1 paper[2] they observed this "thought language mixing" and did some experiments on suppressing it... and the model results got worse.”
“the DeepSeek team explicitly reported this behavior in their R1-zero paper, noting that purely unsupervised reasoning emerges naturally”
“and any modern RL training using synthetic data, e.g. DeepSeek-R1-Zero [1]”
“section 4.2 (first bullet point on PRM) on why this is much trickier to do than it appears.”
“Page 4 of DeepSeek-R1”
“R1 DeepSeek-R1 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models)”