按年被提到
20182018 · 2 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1807.05358 · cs.DC

Beyond Data and Model Parallelism for Deep Neural Networks

Zhihao Jia, Matei Zaharia, Alex Aiken · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-07-31 · 最近 2018-09-19
2不同的人
2活跃月份
2讨论串
297被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The computational requirements for training deep neural networks (DNNs) have grown to the point that it is now standard practice to parallelize training. Existing deep learning systems commonly use data or model parallelism, but unfortunately, these strategies often result in suboptimal parallelization performance. In this paper, we define a more comprehensive search space of parallelization strategies for DNNs called SOAP, which includes strategies to parallelize a DNN in the Sample, Operation, Attribute, and Parameter dimensions. We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine. To accelerate this search, FlexFlow introduces a novel execution simulator that can accurately predict a parallelization strategy's performance and is three orders of magnitude faster than prior approaches that have to execute each strategy. We evaluate FlexFlow with six real-world DNN benchmarks on two GPU clusters and show that FlexFlow can increase training throughput by up to 3.8x over state-of-the-art approaches, even when including its search time, and also improves scalability.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论eslaught · 2018-09

“Wow, this is a tough set of questions! Some initial thoughts: > How does Legion differ from languages like Halide which separate scheduling from the algorithm? Yes, that's one of the similarities. Legion, Sequoia, and Halide all share a separation between the specification of the algorithm from the mapping to any particular architecture. The biggest di…”

David Patterson Says It’s Time for New Computer Architectures and Languages原评论 ↗
被提交为 story · 1 次