“I highly recommend the paper "Understanding deep learning requires rethinking generalization"”
Understanding deep learning requires rethinking generalization
大网络能拟合随机标签,传统泛化理论因此失效
论文通过大量系统实验表明,用随机梯度方法训练的最先进图像分类卷积网络可以轻松拟合随机标签,甚至拟合完全无结构的随机噪声,而且显式正则化几乎不影响这一现象,因此传统上归因于模型族性质或正则化的解释无法说明大网络为何泛化良好。作者还给出理论构造:只要参数数量超过数据点数,简单的两层网络就具有完美的有限样本表达能力。
HN 上的人把它当作开启深度学习泛化研究的里程碑,常用来说明网络能记住噪声却仍能泛化、经典的容量理论已不够用,并引出隐式正则化等后续工作。
学习路径
读这篇之前,按顺序读书架上的 6 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径建立 ImageNet 上的主流架构、正则化手段和损失曲面的背景,再读该论文如何用随机标签实验挑战传统泛化观。
- 1
论文的核心实验在 ImageNet 与 CIFAR 一类图像分类任务上进行,需要先理解该基准。
读抓任务定义和评价指标。
- 2
dropout 是论文逐个检验的显式正则化手段之一。
读抓 dropout 的做法与它为什么能减少过拟合。
- 3
批归一化被论文视为隐式正则化的例子来检验。
读抓归一化公式及它对训练的影响。
- 4
Inception 系列架构是论文实验使用的网络之一。
读抓分解卷积的思路和标签平滑一类正则化;结构细节略读。
- 5
残差网络是现代图像分类的标准架构,论文在这类大网络上演示随机标签拟合。
读抓残差块的结构,理解为什么大网络容量很大。
- 6
从损失曲面的角度讨论大网络的局部极小值,对应论文关于优化与泛化关系的讨论。
读抓球形自旋玻璃模型得出的结论:大网络局部极小值质量相近。
- 7Understanding deep learning requires rethinking generalization
大网络能拟合随机标签,传统泛化理论因此失效。
读抓随机标签、随机像素等实验设计,以及对容量和显式、隐式正则化的讨论。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Check out a recent paper that explores exactly this questions:”
“Understanding deep learning requires rethinking generalization is landmark for kicking off this whole generalization business”
“Check out this classic paper on deep learning with randomized labels:”
“I think you'd be interested in Understanding deep learning requires rethinking generalization. It discusses how SGD is an implicit regularizer.”
“And in all likelihood the neural net will learn that noise: Understanding deep learning requires rethinking generalization”
看全部 16 条 →
“there is some evidence that the structural priors in complex deep networks actually inhibit overfitting and improve generalization [1].”
“In fact recent research indicates that you can randomly relabel the training examples and the network still achieves zero training error.”
“we don't even know why they generalise in the first place! See: Understanding deep learning requires rethinking generalization”
“That paper was really famous. It showed you can get zero training loss on data when you replace the labels with random noise.”
“This was a huge paper a couple years ago that demonstrates deep nets will still find the structure in the data even with totally randomized labels: Understanding deep learning requires rethinking generalization”
“There exist weights that perform perfectly on the training data but give random answers on the test data [0].”
“The question then becomes: why does this generalize [4], given that the classical theory of Vapnik and others [5] becomes vacuous, no longer guaranteeing lack of over-fitting?”
“[1] Was an empirical paper that inspired much theoretical follow-up.”
“The paper [0] noted a particularly odd feature of neural net training: one can train neural nets on standard datasets to fit random labels.”
“What Zhang et al [2] showed convincingly in the Rethinking paper is that just focusing on the hypothesis space cannot be enough”
- Understanding deep learning requires rethinking generalization ↗visionscaper · 2017-02-04 · 131 分
- Understanding deep learning requires rethinking generalization ↗sytelus · 2018-12-15 · 6 分
- Understanding Deep Learning Requires Re-Thinking Generalization (2017) ↗tosh · 2020-08-12 · 2 分
- Understanding deep learning requires rethinking generalization ↗sytelus · 2020-09-17 · 2 分