按年被提到
20192020 · 2 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1911.11134 · cs.LG

Rigging the Lottery: Making All Tickets Winners

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, Erich Elsen · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2020-03-07 · 最近 2026-04-24
3不同的人
2活跃月份
3讨论串
210被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and datasets, including ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static. Code used in our work can be found in github.com/google-research/rigl.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论ekelsen · 2020-03

“The implications are unclear to me. We already know how to prune models for inference. For example https://arxiv.org/abs/1710.01878, along with earlier work (and more recent work). There's also work showing that you can take advantage of the sparsity to achieve practical speed gains: https://arxiv.org/abs/1911.09723. We can also train networks that are s…”

Proving the Lottery Ticket Hypothesis: Pruning is All You Need原评论 ↗
评论DARSHANFOFADIYA · 2026-04

“Most "sparse training" in PyTorch today isn't actually sparse. A binary mask gets multiplied into a dense weight matrix, which means the zeros still consume memory, still move through the cache, and still get multiplied. That's pruning simulation, not sparse computation. SparseLab does the other thing: real sparse storage (a custom Padded-CSR layout), cus…”

Show HN: SparseLab–real sparse training(CSR+custom kernel) in PyTorch, CPU-first原评论 ↗
被提交为 story · 1 次