按年被提到
20192019 · 1 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1902.09574 · cs.LG

The State of Sparsity in Deep Neural Networks

Trevor Gale, Erich Elsen, Sara Hooker · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-02-28 · 最近 2020-12-20
2不同的人
2活跃月份
2讨论串
549被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论sdenton4 · 2020-12

“(Lottery Ticket, to date, produces small networks ex post facto... You still have to train the giant network. There's also some indication that it's chancy on 'large' datasets+problems. https://arxiv.org/abs/1902.09574 )”

Researchers propose faster, more efficient alternatives to backpropagation原评论 ↗
被提交为 story · 1 次