“https://arxiv.org/abs/1803.05407 "Averaging Weights Leads to Wider Optima and Better Generalization" Weirdly, the averaging doesn't have to be synchronous.”
Averaging Weights Leads to Wider Optima and Better Generalization
Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“From the Related Work section (best guess): Stochastic Weight Averaging (Izmailov et al 2018) https://arxiv.org/abs/1803.05407 Latest Weight Averaging (Kaddour 2022) https://arxiv.org/abs/2209.14981 Latest Weight Averaging? (Sanyal et al 2023) https://arxiv.org/abs/2311.16294 Cyclic Learning Rates (Portes et al 2022) https://arxiv.org/abs/2206.00832 …”