按年被提到
20182018 · 1 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:1811.03962 · cs.LG

A Convergence Theory for Deep Learning via Over-Parameterization

Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-11-14 · 最近 2023-01-19
2不同的人
2活跃月份
2讨论串
771被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works has been focusing on training neural networks with one hidden layer. The theory of multi-layer networks remains largely unsettled. In this work, we prove why stochastic gradient descent (SGD) can find $\textit{global minima}$ on the training objective of DNNs in $\textit{polynomial time}$. We only make two assumptions: the inputs are non-degenerate and the network is over-parameterized. The latter means the network width is sufficiently large: $\textit{polynomial}$ in $L$, the number of layers and in $n$, the number of samples. Our key technique is to derive that, in a sufficiently large neighborhood of the random initialization, the optimization landscape is almost-convex and semi-smooth even with ReLU activations. This implies an equivalence between over-parameterized neural networks and neural tangent kernel (NTK) in the finite (and polynomial) width setting. As concrete examples, starting from randomly initialized weights, we prove that SGD can attain 100% training accuracy in classification tasks, or minimize regression loss in linear convergence speed, with running time polynomial in $n,L$. Our theory applies to the widely-used but non-smooth ReLU activation, and to any smooth and possibly non-convex loss functions. In terms of network architectures, our theory at least applies to fully-connected neural networks, convolutional neural networks (CNN), and residual neural networks (ResNet).

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论gogut · 2018-11

“This paper appears in November, but in fact, Allen-Zhu (MSR, http://people.csail.mit.edu/zeyuan/ ) already posted his result in Oct. This is their first paper in Oct:https://arxiv.org/pdf/1810.12065.pdf, this is their second paper in Nov https://arxiv.org/pdf/1811.03962.pdf. In MSR Oct paper, they proved how to train RNN (which is even harder than DNN). …”

Gradient Descent Finds Global Minima of Deep Neural Networks原评论 ↗
评论kvathupo · 2023-01

“I think the battle to promote the use of quantum computers is rendered even more difficult by the fact that their raison d'être of "being faster than classical computers" requires one to further develop the existing field of complexity analysis for classical computers. Indeed, it's difficult to fairly compare the gate/oracle-based complexity of quantum co…”

IBM Quantum System Two原评论 ↗