按年被提到
20192019 · 1 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1910.00762 · cs.LG

Accelerating Deep Learning by Focusing on the Biggest Losers

Angela H. Jiang, Daniel L. -K. Wong, Giulio Zhou, David G. Andersen, Jeffrey Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C. Lipton, Padmanabhan Pillai · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-10-07 · 最近 2022-05-18
2不同的人
2活跃月份
2讨论串
79被引(OpenAlex)
arXiv 摘要(候选层不写导读)

This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update parameters, or to skip immediately to the next example. By reducing the number of computationally-expensive backpropagation steps performed, Selective-Backprop accelerates training. Evaluation on CIFAR10, CIFAR100, and SVHN, across a variety of modern image models, shows that Selective-Backprop converges to target error rates up to 3.5x faster than with standard SGD and between 1.02--1.8x faster than a state-of-the-art importance sampling approach. Further acceleration of 26% can be achieved by using stale forward pass results for selection, thus also skipping forward passes of low priority examples.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论dgacmu · 2022-05

“You have to stash more information from the forward pass in order to calculate the gradients during backprop. You can't just naively use an inference accelerator as part of training - inference-only gets to discard intermediate activations immediately. (Also, many inference accelerators use lower precision than you do when training) There are tricks you…”

Accelerated PyTorch Training on M1 Mac原评论 ↗
被提交为 story · 1 次