按年被提到
20162017 · 5 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:1611.03530 · cs.LG

Understanding deep learning requires rethinking generalization

理解深度学习需要重新思考泛化
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals · 2016

大网络能拟合随机标签,传统泛化理论因此失效

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 18 人首次 2016-12-27 · 最近 2025-03-19发表于 Communications of the ACM
18不同的人
18活跃月份
18讨论串
2,837被引(OpenAlex)

论文通过大量系统实验表明,用随机梯度方法训练的最先进图像分类卷积网络可以轻松拟合随机标签,甚至拟合完全无结构的随机噪声,而且显式正则化几乎不影响这一现象,因此传统上归因于模型族性质或正则化的解释无法说明大网络为何泛化良好。作者还给出理论构造:只要参数数量超过数据点数,简单的两层网络就具有完美的有限样本表达能力。

HN 为什么提它

HN 上的人把它当作开启深度学习泛化研究的里程碑,常用来说明网络能记住噪声却仍能泛化、经典的容量理论已不够用,并引出隐式正则化等后续工作。

实证研究难度 3 / 5约 60 分钟经得起时间
读之前最好知道
1统计学习理论与泛化误差
2VC 维与模型容量的基本概念
3随机梯度下降(SGD)与正则化
4深度神经网络基础

学习路径

读这篇之前,按顺序读书架上的 6 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出

这条路径建立 ImageNet 上的主流架构、正则化手段和损失曲面的背景,再读该论文如何用随机标签实验挑战传统泛化观。

  1. 1
    ImageNet Large Scale Visual Recognition ChallengeImageNet 大规模视觉识别挑战 · 2014

    论文的核心实验在 ImageNet 与 CIFAR 一类图像分类任务上进行,需要先理解该基准。

    读抓任务定义和评价指标。

  2. 2
    Improving neural networks by preventing co-adaptation of feature detectors通过防止特征检测器的共适应来改进神经网络 · 2012

    dropout 是论文逐个检验的显式正则化手段之一。

    读抓 dropout 的做法与它为什么能减少过拟合。

  3. 3
    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift批归一化:通过减少内部协变量偏移加速深度网络训练 · 2015

    批归一化被论文视为隐式正则化的例子来检验。

    读抓归一化公式及它对训练的影响。

  4. 4
    Rethinking the Inception Architecture for Computer Vision重新思考计算机视觉中的 Inception 架构 · 2015

    Inception 系列架构是论文实验使用的网络之一。

    读抓分解卷积的思路和标签平滑一类正则化;结构细节略读。

  5. 5
    Deep Residual Learning for Image Recognition用于图像识别的深度残差学习 · 2015

    残差网络是现代图像分类的标准架构,论文在这类大网络上演示随机标签拟合。

    读抓残差块的结构,理解为什么大网络容量很大。

  6. 6
    The Loss Surfaces of Multilayer Networks多层网络的损失曲面 · 2014

    从损失曲面的角度讨论大网络的局部极小值,对应论文关于优化与泛化关系的讨论。

    读抓球形自旋玻璃模型得出的结论:大网络局部极小值质量相近。

  7. 7
    Understanding deep learning requires rethinking generalization理解深度学习需要重新思考泛化 · 2016 · 就是这篇

    大网络能拟合随机标签,传统泛化理论因此失效。

    读抓随机标签、随机像素等实验设计,以及对容量和显式、隐式正则化的讨论。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
强烈推荐rockmeamedee · 2018-07

“I highly recommend the paper "Understanding deep learning requires rethinking generalization"”

强烈推荐这篇论文,包括 RNN 对纯噪声的记忆能力
Teaching Computers Art History原评论 ↗
推荐jureso · 2017-01

“Check out a recent paper that explores exactly this questions:”

推荐这篇论文,它正好探讨大网络为何在实践中不过拟合
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer原评论 ↗
推荐chillee · 2017-12

“Understanding deep learning requires rethinking generalization is landmark for kicking off this whole generalization business”

称这篇是开启泛化研究热潮的里程碑,说明基于容量的经典理论解释不了神经网络
An Interview with an Anonymous Data Scientist (2016)原评论 ↗
推荐sdenton4 · 2018-10

“Check out this classic paper on deep learning with randomized labels:”

推荐这篇经典论文,展示网络能拟合随机标签
The Mixed Track Record of Machine Learning Algorithms原评论 ↗
推荐machinelearning · 2020-10

“I think you'd be interested in Understanding deep learning requires rethinking generalization. It discusses how SGD is an implicit regularizer.”

推荐给对方,它讨论了 SGD 如何起到隐式正则化的作用
Why deep learning works even though it shouldn’t原评论 ↗
引用highd · 2016-12

“And in all likelihood the neural net will learn that noise: Understanding deep learning requires rethinking generalization”

指出神经网络很可能把噪声也学进去,所以需要留出测试集
LSTM Neural Networks for Time Series Prediction原评论 ↗
看全部 16 条 →
引用backpropaganda · 2017-05

“there is some evidence that the structural priors in complex deep networks actually inhibit overfitting and improve generalization [1].”

引用它,称有证据表明复杂深度网络的结构先验反而抑制过拟合
Deep Learning’s Impact on Image Processing, Mathematics, and Humanity原评论 ↗
引用hyperbovine · 2017-08

“In fact recent research indicates that you can randomly relabel the training examples and the network still achieves zero training error.”

指出随机重标训练样本后网络仍能达到零训练误差
Why Deep Learning surprises me原评论 ↗
引用eggie5 · 2018-01

“we don't even know why they generalise in the first place! See: Understanding deep learning requires rethinking generalization”

感叹我们甚至不知道网络为何能泛化,并引用这篇论文
Cloud AutoML: Making AI accessible to every business原评论 ↗
引用skeptic_69 · 2018-11

“That paper was really famous. It showed you can get zero training loss on data when you replace the labels with random noise.”

称这篇论文很有名,它表明把标签换成随机噪声也能得到零训练损失
Gradient Descent Finds Global Minima of Deep Neural Networks原评论 ↗
引用jasonwatkinspdx · 2020-02

“This was a huge paper a couple years ago that demonstrates deep nets will still find the structure in the data even with totally randomized labels: Understanding deep learning requires rethinking generalization”

称这是影响巨大的论文,表明深度网络即使标签完全随机仍能拟合数据
A popular self-driving car dataset is missing labels for hundreds of pedestrians原评论 ↗
引用woopwoop · 2022-05

“There exist weights that perform perfectly on the training data but give random answers on the test data [0].”

引用它说明存在在训练数据上完美、在测试数据上却随机作答的权重
Learning to think critically about machine learning原评论 ↗
引用DavidSJ · 2022-05

“The question then becomes: why does this generalize [4], given that the classical theory of Vapnik and others [5] becomes vacuous, no longer guaranteeing lack of over-fitting?”

提出问题:经典学习理论已失效,为什么深度网络还能泛化
Learning to think critically about machine learning原评论 ↗
引用blt · 2024-05

“[1] Was an empirical paper that inspired much theoretical follow-up.”

称它是一篇实证论文,引出了大量理论上的后续工作
Gradient descent visualization原评论 ↗
引用woopwoop · 2025-03

“The paper [0] noted a particularly odd feature of neural net training: one can train neural nets on standard datasets to fit random labels.”

引用它指出神经网络能拟合随机标签这一奇怪特性
Deep Learning Is Not So Mysterious or Different原评论 ↗
引用cgdl · 2025-03

“What Zhang et al [2] showed convincingly in the Rethinking paper is that just focusing on the hypothesis space cannot be enough”

引用它说明只盯着假设空间不足以解释神经网络的泛化
Deep Learning Is Not So Mysterious or Different原评论 ↗
被提交为 story · 4 次