引用pr337h4m · 2023-11
“DiLoCo: Distributed Low-Communication Training of Language Models - DiLoCo”
贴出 DiLoCo,DeepMind 的跨 GPU 集群低通信分布式训练方法
DiLoCo:低通信的分布式语言模型训练,通信量减少 500 倍
标准 LLM 训练要求大量紧密互联的加速器,而论文提出分布式低通信优化算法 DiLoCo,可在互联较差的设备“孤岛”上训练语言模型。它是联邦平均的变体,内层步数很大、内层优化器为 AdamW、外层优化器为 Nesterov 动量。在 C4 上,8 个 worker 的 DiLoCo 与完全同步优化效果相当,而通信量少 500 倍;对各 worker 的数据分布和资源增减都很鲁棒。
评论者把它当作能跨互联网或跨集群训练模型的低通信方法来引用,并提到它被产品化、成为 Intellect-2 的基础。
“DiLoCo: Distributed Low-Communication Training of Language Models - DiLoCo”
“There has been some interesting work when it comes to distributed training. For example DiLoCo.”
“There are methods that make it feasible to train models over the internet. DiLoCo is one [1]”
“The most interesting thing I see is the productization of the diloco work done here [1].”