按年被提到
20252025 · 18 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2501.12948 · cs.CL

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-R1:通过强化学习激励大模型的推理能力
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang 等 200 人 · 2025

纯强化学习即可激发大模型推理,无需人工标注的推理轨迹

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 22 人首次 2025-01-23 · 最近 2026-07-12
22不同的人
9活跃月份
20讨论串
5,548被引(OpenAlex)

论文表明,大语言模型的推理能力可以通过纯强化学习(RL)激发出来,无需人工标注的推理轨迹。该 RL 框架促成自我反思、验证和动态调整策略等高级推理模式的涌现,所训练的模型在数学、编程竞赛和 STEM 等可验证任务上优于基于人类示范做传统监督学习的对照模型;这些涌现的推理模式还可被系统地用来增强较小模型的推理能力。

HN 为什么提它

评论者把它当作理解 DeepSeek 如何以较少算力得到强推理模型的起点,常引用其中的 GRPO、「顿悟时刻」、失败尝试(MCTS、PRM)、语言混用以及蒸馏小模型等细节。

方法难度 4 / 5约 90 分钟经得起时间
读之前最好知道
1大语言模型与思维链(chain-of-thought)
2强化学习(PPO、GRPO)
3可验证奖励与奖励模型
4监督微调与知识蒸馏

学习路径

读这篇之前,按顺序读书架上的 3 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出

先搞清 R1 用的强化学习算法(PPO 到 GRPO)和它所基于的 DeepSeek-V3 基座,再读 R1 如何用纯 RL 激发推理。

  1. 1

    GRPO 是 PPO 的变体,先要懂 PPO 的策略梯度和裁剪目标。

    读clipped surrogate objective 与优势估计(advantage)的概念;跳过 Atari 与机器人实验细节。

  2. 2
    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsDeepSeekMath:推高开源语言模型的数学推理极限 · 2024

    R1 的 RL 算法 GRPO 正是在这篇里提出。

    读GRPO 一节:用组内相对奖励代替价值网络(critic)做基线;对比 PPO 的差别。

  3. 3
    DeepSeek-V3 Technical ReportDeepSeek-V3 技术报告 · 2024

    DeepSeek-R1 的基座模型 R1-Zero 与 R1 都建立在 V3 之上。

    读只看模型架构概览(MoE、MLA)和后训练部分;训练基础设施可略读。

  4. 4
    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-R1:通过强化学习激励大模型的推理能力 · 2025 · 就是这篇

    目标论文:在上述基座和 GRPO 之上,用规则奖励做纯 RL,得到长思维链推理。

    读R1-Zero 的奖励设计与训练过程中推理长度增长、顿悟时刻;再看冷启动数据和多阶段流程。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐verdverm · 2025-01

“There are some interesting contributions in the paper too.”

认为论文也有一些有趣的贡献,模型与 o1 相当且可本地运行
DeepSeek temporarily limits registration to Chinese mobile numbers原评论 ↗
推荐erulabs · 2025-01

“The paper for R1 (1) seems to be pretty clear how they got such good results with so little horsepower (see: 'Group Relative Policy Optimization').”

认为论文清楚说明了如何靠 GRPO 用很少算力取得好结果
DeepSeek releases Janus Pro, a text-to-image generator [pdf]原评论 ↗
推荐desideratum · 2026-04

“Many recent OSS models have great tech reports where you can learn more about these kind of things:”

推荐阅读 R1 技术报告,了解用 RL 激励推理的训练方法
Nanocode: The best Claude Code that $200 can buy in pure JAX on TPUs原评论 ↗
引用lhl · 2025-01

“Here is the R1 paper: DeepSeek-R1”

给出 R1 论文,作为了解 DeepSeek 训练创新的起点
DeepSeek and the Effects of GPU Export Controls原评论 ↗
引用attentionmech · 2025-01

“If you check failure section of their paper, they also tried other methods like MCTS and PRM which is what other labs have been obsessing about but couldn't move on from (that includes bigshots).”

指出论文失败部分试过 MCTS 和 PRM 但未采用,而其他实验室一直执着于此
Emerging reasoning with reinforcement learning原评论 ↗
引用shihab · 2025-01

“Link [2] to the result on more standard LLM benchmarks. They conveniently placed the results on the first page of the paper.”

指出 R1 的标准基准结果就在论文第一页
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL原评论 ↗
看全部 20 条 →
引用highfrequency · 2025-01

“The R1 paper emphasizes their success with reinforcement learning without requiring any supervised data (unlike RLHF for example).”

指出 R1 论文强调不需要监督数据,纯靠 RL 取得成功
The impact of competition and DeepSeek on Nvidia原评论 ↗
引用throwup238 · 2025-01

“it’s explicitly spelled out in DeepSeek’s R1 paper that they had trouble with improving over DeepSeek-V3”

指出论文明说在软件工程任务上难以超越 DeepSeek-V3
The Illustrated DeepSeek-R1原评论 ↗
引用btown · 2025-01

“and DeepSeek-R1 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps.”

指出 R1 基于 V3-Base,但似乎没给出 RL 步骤的成本
Berkeley Researchers Replicate DeepSeek R1's Core Tech for Just $30: A Small Mod原评论 ↗
引用fulladder · 2025-01

“Zero evidence that the above statement is true, and weak evidence (authors' claims) that it is false. Have you read their papers even?”

拿 R1 论文作弱反证,反驳 $1B+ 预算的说法
Exposed DeepSeek database leaking sensitive information, including chat history原评论 ↗
引用cadamsdotcom · 2025-01

“DeepSeek talked about “aha moments” in DeepSeek-R1 where under RL the model learned to backtrack and correct itself, for example.”

指出论文讲了 RL 下模型学会回溯和自我纠正的 aha moments
Cognitive Reasoning Agents and the Extended Information Filter原评论 ↗
引用dingocat · 2025-01

“DeepSeek released a distilled version based on a 1.5B Qwen model with the full name DeepSeek-R1-Distill-Qwen-1.5B, see chapter 3.2 on page 14 of their research article [0].”

指出论文 3.2 节介绍了蒸馏版模型,如 Distill-Qwen-1.5B
Llama.cpp supports Vulkan. why doesn't Ollama?原评论 ↗
引用alexhutcheson · 2025-02

“DeepSeek-R1 had an RLHF step in their post-training pipeline (section 2.3.4 of their technical report[1]).”

指出技术报告 2.3.4 节显示 R1 的后训练流程里有 RLHF 步骤
RLHF Book原评论 ↗
引用helloplanets · 2025-06

“[0]: [LINK] [1]: DeepSeek-R1 [2]: [LINK] [3]: [LINK]”

The Gentle Singularity原评论 ↗
引用mindcrime · 2025-06

“in the DeepSeek R1 paper[2] they observed this "thought language mixing" and did some experiments on suppressing it... and the model results got worse.”

指出 R1 论文观察到思考语言混用,抑制后模型结果反而变差
Ask HN: Can anybody clarify why OpenAI reasoning now shows non-English thoughts?原评论 ↗
引用diwank · 2025-06

“the DeepSeek team explicitly reported this behavior in their R1-zero paper, noting that purely unsupervised reasoning emerges naturally”

指出 R1-zero 论文已报告纯无监督推理下自然出现语言混用
Ask HN: Can anybody clarify why OpenAI reasoning now shows non-English thoughts?原评论 ↗
引用charleshn · 2025-09

“and any modern RL training using synthetic data, e.g. DeepSeek-R1-Zero [1]”

把 DeepSeek-R1-Zero 作为用合成数据做 RL 训练的例子
The Bitter Lesson Is Misunderstood原评论 ↗
引用thegeomaster · 2025-11

“section 4.2 (first bullet point on PRM) on why this is much trickier to do than it appears.”

指出 4.2 节解释了为什么 process reward model 比看起来更难做
The inefficiency of RL, and implications for RLVR progress原评论 ↗
引用nl · 2026-01

“Page 4 of DeepSeek-R1”

指向论文第 4 页,关于用编译器反馈做 GRPO 的代码竞赛提示
Nanolang: A tiny experimental language designed to be targeted by coding LLMs原评论 ↗
引用gobdovan · 2026-06

“R1 DeepSeek-R1 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models)”

指出 RL 早已有之,但 R1 公开地在大模型上大规模展示了它
How LLMs Work原评论 ↗