“A. Krizhevsky put out a pretty interesting paper recently on this same topic.”
One weird trick for parallelizing convolutional neural networks
卷积层用数据并行、全连接层用模型并行的多 GPU 训练法
论文提出一种在多块 GPU 上并行训练卷积神经网络的新方法,在应用于现代卷积网络时,其扩展性明显优于所有其他方案。从 HN 的讨论看,其做法是对卷积层采用数据并行、对全连接层采用模型并行的混合并行。
HN 上的人把它当作解释数据并行与模型并行区别的清晰范例,也在谈多 GPU 训练时感叹这个窍门一直没有进入主流框架。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“For anyone actually interested in some interesting techniques for multi-GPU DNN training, One weird trick for parallelizing convolutional neural networks and references therein are probably a good start.”
“A good explanation is the One Weird Trick paper: One weird trick for parallelizing convolutional neural networks”
“an approach which allows an 8-GPU Big Sur with 12.5GB/s P2P bandwidth to go toe to toe with an 8-GPU DGX-1 on training AlexNet.”
“This scheme is much convoluted but very effective when training large MLP.”
“nice, achieved 6x speed up on an 8 gpu box”
看全部 10 条 →
“One weird trick for parallelizing ConvNets: One weird trick for parallelizing convolutional neural networks”
“Everyone has been waiting for that day since One Weird Trick[1].”
“I continue to be boggled that Alex Krizhevsky's One Weird Trick never made it to TensorFlow or anywhere else:”
“Looks like what it does is similar to what Alex did a few years back with One Weird Trick paper: One weird trick for parallelizing convolutional neural networks”
- One weird trick for parallelizing convolutional neural networks ↗signa11 · 2014-07-24 · 3 分