“Hinton et. al. wrote this quite approachable paper on the topic:”
Distilling the Knowledge in a Neural Network
把模型集成的知识压缩进单个小模型:知识蒸馏
训练多个模型再平均预测能提升效果,但部署整个集成代价高昂。论文沿着 Caruana 等人把集成知识压缩进单个模型的思路,用另一种压缩技术做了进一步发展,在 MNIST 上取得意外好的结果,并显著改进了一个大规模商用系统的声学模型。作者还提出由完整模型加多个专家模型组成的新型集成,专家模型可快速并行训练,用来区分完整模型容易混淆的细粒度类别。
评论者把它当作知识蒸馏的起源论文来引用,强调蒸馏早于 LLM、是用大模型输出训练小模型的经典做法,也有人称它写得相当易读。
学习路径
读这篇之前,按顺序读书架上的 1 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出知识蒸馏与集成、dropout 这类隐式集成的思路相关,先读 dropout,再读蒸馏。
- 1
dropout 可看作训练指数多个子网络的隐式集成,帮助理解为何要把集成压缩进单个模型。
读理解随机丢弃单元如何抑制共适应,以及与模型平均的关系。
- 2Distilling the Knowledge in a Neural Network
用软目标和温度把集成的知识压缩到小模型里。
读抓住带温度的 softmax、软目标与硬目标的结合,以及蒸馏的训练方式。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“A possibly relevant research paper that they didn't mention”
“some form of knowledge distilling [1] (which Hinton came up with, so really no excuse)”
“I think that this one talks about this”
“Distillation compacts neural networks while affecting accuracy minimally.”
“Another approach I've seen that was really cool is Model Distillation [0]”
看全部 17 条 →
“Another approach is to train several different neural networks (an ensemble) on a task, and then train a final neural network based on the average of the ensemble.”
“Distilling the Knowledge in a Neural Network[2] is another approach which leads towards simpler representations.”
“I saw a talk on this paper a couple years ago.”
“It's not nearly at the point yet to be paranoid.”
“which was a precursor to the model distallation work from Geoff Hinton: Distilling the Knowledge in a Neural Network”
“That's a time honoured tradition in ML, invented by the father of the field himself, Geoffrey Hinton, in 2015.”
“Distillation (as you may be aware)”
“The Hinton paper was from 2015 (maybe there is even earlier work):”
“So considering the subject, this is probably the conversion from logits to probabilities as in Hinton's paper Distilling the Knowledge in a Neural Network”
“distillation has been around for 12 years.”
“Distillation a pretty well documented technique that actually pre-dates LLMs Distilling the Knowledge in a Neural Network”
- Distilling the Knowledge in a Neural Network ↗tim_sw · 2015-03-17 · 2 分