按年被提到
20202020 · 1 次2026
机器学习与深度学习›优化与可微编程·arXiv:2011.08895 · cs.LG

ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks

Varun Ranganathan, Alex Lewandowski · 2020
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-11-20 · 最近 2021-03-13
2不同的人
2活跃月份
2讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Gradient descent and backpropagation have enabled neural networks to achieve remarkable results in many real-world applications. Despite ongoing success, training a neural network with gradient descent can be a slow and strenuous affair. We present a simple yet faster training algorithm called Zeroth-Order Relaxed Backpropagation (ZORB). Instead of calculating gradients, ZORB uses the pseudoinverse of targets to backpropagate information. ZORB is designed to reduce the time required to train deep neural networks without penalizing performance. To illustrate the speed up, we trained a feed-forward neural network with 11 layers on MNIST and observed that ZORB converged 300 times faster than Adam while achieving a comparable error rate, without any hyperparameter tuning. We also broaden the scope of ZORB to convolutional neural networks, and apply it to subsamples of the CIFAR-10 dataset. Experiments on standard classification and regression benchmarks demonstrate ZORB's advantage over traditional backpropagation with Gradient Descent.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论The_rationalist · 2021-03

“See also zeroth order backpropagation which allows 300X faster training while not reducing throughput that much https://arxiv.org/abs/2011.08895 How much zero-3 affect accuracy? See also https://github.com/microsoft/fastformers”

Zero-3 Offload: Scale DL models to trillion parameters without code changes原评论 ↗
被提交为 story · 1 次