按年被提到
20182020 · 1 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:1803.05407 · cs.LG

Averaging Weights Leads to Wider Optima and Better Generalization

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-01-09 · 最近 2024-04-06
2不同的人
2活跃月份
2讨论串
567被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论sillysaurusx · 2020-01

“https://arxiv.org/abs/1803.05407 "Averaging Weights Leads to Wider Optima and Better Generalization" Weirdly, the averaging doesn't have to be synchronous.”

An idea from physics helps AI see in higher dimensions原评论 ↗
评论JieJie · 2024-04

“From the Related Work section (best guess): Stochastic Weight Averaging (Izmailov et al 2018) https://arxiv.org/abs/1803.05407 Latest Weight Averaging (Kaddour 2022) https://arxiv.org/abs/2209.14981 Latest Weight Averaging? (Sanyal et al 2023) https://arxiv.org/abs/2311.16294 Cyclic Learning Rates (Portes et al 2022) https://arxiv.org/abs/2206.00832 …”

Schedule-Free Learning – A New Way to Train原评论 ↗