引用Vetch · 2022-11
“See Decentralized Training of Foundation Models in Heterogeneous Environments and their recent GPT-JT test for more.”
指向去中心化训练的研究,并提到 GPT-JT 测试
在慢速异构网络的分散 GPU 上做大模型并行训练的调度方法
论文研究在低带宽、异构网络连接的去中心化 GPU 上,用模型并行训练大型基础模型。核心贡献是一个调度算法,把训练中的不同“tasklet”分配给分散的设备,并给出形式化代价模型和用于寻找最优分配的进化算法。在以真实网络测量模拟的跨 8 个城市、3 个大洲的最极端场景中,作者报告其方法比 Megatron 快 4.8 倍。
评论者把它当作在互联网上分散训练大模型的相关研究,用来回应“能否不靠集中式集群预训练模型”的问题。
“See Decentralized Training of Foundation Models in Heterogeneous Environments and their recent GPT-JT test for more.”
“or [OTHER_PAPER] and Decentralized Training of Foundation Models in Heterogeneous Environments that show you how to do pretraining from scratch in the same setting.”
“See [OTHER_PAPER], Decentralized Training of Foundation Models in Heterogeneous Environments, [OTHER_PAPER] etc:)”