按年被提到
20152017 · 4 次2026
机器学习与深度学习›机器学习实践与反思·arXiv:1503.02531 · stat.ML

Distilling the Knowledge in a Neural Network

蒸馏神经网络中的知识
Geoffrey Hinton, Oriol Vinyals, Jeff Dean · 2015

把模型集成的知识压缩进单个小模型:知识蒸馏

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 18 人首次 2015-03-17 · 最近 2026-08-24
18不同的人
17活跃月份
17讨论串
14,199被引(OpenAlex)

训练多个模型再平均预测能提升效果,但部署整个集成代价高昂。论文沿着 Caruana 等人把集成知识压缩进单个模型的思路,用另一种压缩技术做了进一步发展,在 MNIST 上取得意外好的结果,并显著改进了一个大规模商用系统的声学模型。作者还提出由完整模型加多个专家模型组成的新型集成,专家模型可快速并行训练,用来区分完整模型容易混淆的细粒度类别。

HN 为什么提它

评论者把它当作知识蒸馏的起源论文来引用,强调蒸馏早于 LLM、是用大模型输出训练小模型的经典做法,也有人称它写得相当易读。

奠基之作难度 3 / 5约 30 分钟经得起时间
读之前最好知道
1神经网络与 Softmax
2集成学习(ensemble)
3交叉熵与 KL 散度
4模型压缩

学习路径

读这篇之前,按顺序读书架上的 1 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出

知识蒸馏与集成、dropout 这类隐式集成的思路相关,先读 dropout,再读蒸馏。

  1. 1
    Improving neural networks by preventing co-adaptation of feature detectors通过防止特征检测器的共适应来改进神经网络 · 2012

    dropout 可看作训练指数多个子网络的隐式集成,帮助理解为何要把集成压缩进单个模型。

    读理解随机丢弃单元如何抑制共适应,以及与模型平均的关系。

  2. 2
    Distilling the Knowledge in a Neural Network蒸馏神经网络中的知识 · 2015 · 就是这篇

    用软目标和温度把集成的知识压缩到小模型里。

    读抓住带温度的 softmax、软目标与硬目标的结合,以及蒸馏的训练方式。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐curious_cat_163 · 2022-04

“Hinton et. al. wrote this quite approachable paper on the topic:”

推荐 Hinton 等写的这篇易读的知识蒸馏论文
Scientists discover specific neurons involved in making memories原评论 ↗
引用teraflop · 2015-07

“A possibly relevant research paper that they didn't mention”

指出文章没提到、可能相关的蒸馏论文
How Google Translate squeezes deep learning onto a phone原评论 ↗
引用murbard2 · 2015-07

“some form of knowledge distilling [1] (which Hinton came up with, so really no excuse)”

指出 knowledge distillation 是 Hinton 提出的,没有理由不用
How Google Translate squeezes deep learning onto a phone原评论 ↗
引用gyom · 2016-03

“I think that this one talks about this”

认为这篇谈到了此事,即把集成的预测蒸馏成一个模型
AlphaGo beats Lee Sedol 3-0 [video]原评论 ↗
引用kajecounterhack · 2016-12

“Distillation compacts neural networks while affecting accuracy minimally.”

说明蒸馏能压缩网络且几乎不损失精度
Neurogenesis Deep Learning原评论 ↗
引用iraphael · 2017-02

“Another approach I've seen that was really cool is Model Distillation [0]”

称 model distillation 是一个很酷的方法
How to use Deep Learning when you have Limited Data原评论 ↗
看全部 17 条 →
引用tlb · 2017-04

“Another approach is to train several different neural networks (an ensemble) on a task, and then train a final neural network based on the average of the ensemble.”

描述先训练集成再用其平均结果训练最终网络的做法
The Myth of a Superhuman AI原评论 ↗
引用nl · 2017-08

“Distilling the Knowledge in a Neural Network[2] is another approach which leads towards simpler representations.”

把蒸馏作为得到更简单表示的另一种方法
Inside Darpa’s Push to Make Artificial Intelligence Explain Itself原评论 ↗
引用hidenotslide · 2017-12

“I saw a talk on this paper a couple years ago.”

说自己几年前听过这篇论文的报告
Neural Networks in JavaScript with Deeplearn.js原评论 ↗
引用sovok_x · 2018-02

“It's not nearly at the point yet to be paranoid.”

拿它说明如何把集成蒸馏进一个更小的模型
The crisis of American forensics原评论 ↗
引用FlyingLawnmower · 2020-11

“which was a precursor to the model distallation work from Geoff Hinton: Distilling the Knowledge in a Neural Network”

称模型压缩工作是 Hinton 这篇蒸馏论文的前身
New Explainable AI Algorithms原评论 ↗
引用visarga · 2023-03

“That's a time honoured tradition in ML, invented by the father of the field himself, Geoffrey Hinton, in 2015.”

称蒸馏是 ML 的传统,由 Hinton 于 2015 年发明
Could you train a ChatGPT-beating model for $85,000 and run it in a browser?原评论 ↗
引用etiam · 2024-07

“Distillation (as you may be aware)”

提到 distillation,即训练小模型模仿大模型
Training of Physical Neural Networks原评论 ↗
引用microtonal · 2025-07

“The Hinton paper was from 2015 (maybe there is even earlier work):”

指出 Hinton 2015 年的论文,可能还有更早的工作
Distillation makes AI models smaller and cheaper原评论 ↗
引用arjie · 2026-03

“So considering the subject, this is probably the conversion from logits to probabilities as in Hinton's paper Distilling the Knowledge in a Neural Network”

指出 logits 转概率的 temperature softmax 出自这篇
NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute原评论 ↗
引用dijksterhuis · 2026-07

“distillation has been around for 12 years.”

指出蒸馏技术已有 12 年历史
"We have information that Moonshot distilled Fable for the development of K3"原评论 ↗
引用voxic11 · 2026-08

“Distillation a pretty well documented technique that actually pre-dates LLMs Distilling the Knowledge in a Neural Network”

指出蒸馏是早于 LLM 的成熟技术
OpenAI: GPT 5.6 Sol price reduction (until at least Nov 21)原评论 ↗
被提交为 story · 1 次