按年被提到
20182018 · 1 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:1810.02054 · cs.LG

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Simon S. Du, Xiyu Zhai, Barnabas Poczos, Aarti Singh · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-10-05 · 最近 2021-10-12
2不同的人
2活跃月份
2讨论串
489被引(OpenAlex)
arXiv 摘要(候选层不写导读)

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks. For an $m$ hidden node shallow neural network with ReLU activation and $n$ training data, we show as long as $m$ is large enough and no two inputs are parallel, randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. Our analysis relies on the following observation: over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations, which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum. We believe these insights are also useful in analyzing deep models and other first order methods.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论leecarraher · 2021-10

“They also have an embedded link to this paper(https://arxiv.org/abs/1810.02054 ) regarding the provable convergence of neural networks by non-convex gradient descent optimization. You are correct, the researchers do no propose an alternative to sgd based training (or at least according to the article). It did seem like it was going toward an alternative t…”

A new link to an old model could crack the mystery of deep learning原评论 ↗
被提交为 story · 1 次