按年被提到
20192019 · 1 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:1903.01611 · cs.LG

Stabilizing the Lottery Ticket Hypothesis

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-05-07 · 最近 2020-05-08
2不同的人
2活跃月份
2讨论串
184被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket hypothesis" conjectures that typical neural networks contain small subnetworks that can train to similar accuracy in a commensurate number of steps. The evidence for this claim is that a procedure based on iterative magnitude pruning (IMP) reliably finds such subnetworks retroactively on small vision tasks. However, IMP fails on deeper networks, and proposed methods to prune before training or train pruned networks encounter similar scaling limitations. In this paper, we argue that these efforts have struggled on deeper networks because they have focused on pruning precisely at initialization. We modify IMP to search for subnetworks that could have been obtained by pruning early in training (0.1% to 7% through) rather than at iteration 0. With this change, it finds small subnetworks of deeper networks (e.g., 80% sparsity on Resnet-50) that can complete the training process to match the accuracy of the original network on more challenging tasks (e.g., ImageNet). In situations where IMP fails at iteration 0, the accuracy benefits of delaying pruning accrue rapidly over the earliest iterations of training. To explain these behaviors, we study subnetwork "stability," finding that - as accuracy improves in this fashion - IMP subnetworks train to parameters closer to those of the full network and do so with improved consistency in the face of gradient noise. These results offer new insights into the opportunity to prune large-scale networks early in training and the behaviors underlying the lottery ticket hypothesis

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论mkolodny · 2020-05

“Not quite. The Lottery Ticket Hypothesis paper showed that models could shrink to around 10% of their original size without a loss of accuracy [0]. So around a million neurons instead of 10 million. [0] https://arxiv.org/abs/1903.01611v1”

Researchers unveil a pruning algorithm to shrink deep learning models原评论 ↗
被提交为 story · 1 次