按年被提到
20182023 · 1 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1802.04434 · cs.LG

signSGD: Compressed Optimisation for Non-Convex Problems

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, Anima Anandkumar · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-02-18 · 最近 2024-10-12
2不同的人
2活跃月份
2讨论串
540被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Training large neural networks requires distributing learning across multiple workers, where the cost of communicating gradients can be a significant bottleneck. signSGD alleviates this problem by transmitting just the sign of each minibatch stochastic gradient. We prove that it can get the best of both worlds: compressed gradients and SGD-level convergence rate. The relative $\ell_1/\ell_2$ geometry of gradients, noise and curvature informs whether signSGD or SGD is theoretically better suited to a particular problem. On the practical side we find that the momentum counterpart of signSGD is able to match the accuracy and convergence speed of Adam on deep Imagenet models. We extend our theory to the distributed setting, where the parameter server uses majority vote to aggregate gradient signs from each worker enabling 1-bit compression of worker-server communication in both directions. Using a theorem by Gauss we prove that majority vote can achieve the same reduction in variance as full precision distributed SGD. Thus, there is great promise for sign-based optimisation schemes to achieve fast communication and fast convergence. Code to reproduce experiments is to be found at https://github.com/jxbz/signSGD .

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论mtcrawshaw · 2023-02

“Your formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will change the value of the momentum variable for subsequent steps. To be fair, I don't think that this operation will fundamentally change the behavior of the algorithm. In fact, I don't think their algor…”

Symbolic Discovery of Optimization Algorithms原评论 ↗
评论f_devd · 2024-10

“It has been more formally studied in signSGD[0], and empirically it's comparable to Adam in terms of behavior. [0]: https://arxiv.org/pdf/1802.04434”

INTELLECT–1: Launching the First Decentralized Training of a 10B Parameter Model原评论 ↗