按年被提到
20182018 · 2 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1811.03600 · cs.LG

Measuring the Effects of Data Parallelism on Neural Network Training

Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-11-09 · 最近 2018-12-12
2不同的人
2活跃月份
2讨论串
229被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论antognini · 2018-12

“I worked on a research project that set out to answer that question: https://arxiv.org/abs/1811.03600 We looked at a bunch of different model architectures and datasets and found that you can get speedups from larger batch sizes up until a certain point, but that point is different for different datasets and architectures. The range of good hyperparamet…”

Train TensorFlow models faster and at lower cost on Cloud TPU Pods原评论 ↗
被提交为 story · 1 次