按年被提到
20182018 · 1 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1802.09941 · cs.LG

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

Tal Ben-Nun, Torsten Hoefler · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-02-28 · 最近 2020-08-18
2不同的人
2活跃月份
2讨论串
204被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论mattlutze · 2020-08

“Distributed training is indeed a thing! A few random Arxiv pulls: https://arxiv.org/abs/2007.03970 https://arxiv.org/abs/1802.09941 Efficiency and resource cost is the big question though. You don't pay for the electricity or part wear that you don't use, and home computers or workstations may not be as efficient at performing a training run vs a task-s…”

Karpathy's MinGPT原评论 ↗
被提交为 story · 1 次