按年被提到
20212023 · 2 次2026
机器学习与深度学习›机器学习实践与反思·arXiv:2104.05158 · cs.DC

Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models

Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, Jie Amy Yang 等 53 人 · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2022-01-24 · 最近 2023-05-10
2不同的人
3活跃月份
3讨论串
157被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Deep learning recommendation models (DLRMs) are used across many business-critical services at Facebook and are the single largest AI application in terms of infrastructure demand in its data-centers. In this paper we discuss the SW/HW co-designed solution for high-performance distributed training of large-scale DLRMs. We introduce a high-performance scalable software stack based on PyTorch and pair it with the new evolution of Zion platform, namely ZionEX. We demonstrate the capability to train very large DLRMs with up to 12 Trillion parameters and show that we can attain 40X speedup in terms of time to solution over previous systems. We achieve this by (i) designing the ZionEX platform with dedicated scale-out network, provisioned with high bandwidth, optimal topology and efficient transport (ii) implementing an optimized PyTorch-based training stack supporting both model and data parallelism (iii) developing sharding algorithms capable of hierarchical partitioning of the embedding tables along row, column dimensions and load balancing them across multiple workers; (iv) adding high-performance core operators while retaining flexibility to support optimizers with fully deterministic updates (v) leveraging reduced precision communications, multi-level memory hierarchy (HBM+DDR+SSD) and pipelining. Furthermore, we develop and briefly comment on distributed data ingestion and other supporting services that are required for the robust and efficient end-to-end training in production environments.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论macrolocal · 2022-01

“Hm, maybe they're optimizing DLRM models, since inefficient HW communication sometimes bottlenecks Facebook's data center performance for them [1]. The improvements would be better personalization, ie. monetization. [1] https://arxiv.org/pdf/2104.05158.pdf”

Meta Unveils New AI Supercomputer原评论 ↗
被提交为 story · 2 次