按年被提到
20142014 · 4 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:1404.5997 · cs.NE

One weird trick for parallelizing convolutional neural networks

并行化卷积神经网络的一个古怪窍门
Alex Krizhevsky · 2014

卷积层用数据并行、全连接层用模型并行的多 GPU 训练法

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 10 人首次 2014-04-28 · 最近 2022-06-09
10不同的人
11活跃月份
11讨论串
952被引(OpenAlex)

论文提出一种在多块 GPU 上并行训练卷积神经网络的新方法,在应用于现代卷积网络时,其扩展性明显优于所有其他方案。从 HN 的讨论看,其做法是对卷积层采用数据并行、对全连接层采用模型并行的混合并行。

HN 为什么提它

HN 上的人把它当作解释数据并行与模型并行区别的清晰范例,也在谈多 GPU 训练时感叹这个窍门一直没有进入主流框架。

方法难度 3 / 5约 20 分钟经得起时间
读之前最好知道
1卷积神经网络(CNN)基础
2数据并行与模型并行(data/model parallelism)
3GPU 训练与多卡通信
4随机梯度下降(SGD)的小批量训练

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐kastnerkyle · 2014-04

“A. Krizhevsky put out a pretty interesting paper recently on this same topic.”

推荐 Krizhevsky 这篇论文,按层类型在多 GPU 间拆分 CNN 并行
Data Scientist Interviews原评论 ↗
推荐ajtulloch · 2014-11

“For anyone actually interested in some interesting techniques for multi-GPU DNN training, One weird trick for parallelizing convolutional neural networks and references therein are probably a good start.”

推荐给想了解多 GPU 深度网络训练技术的人,作为入门起点
Distributing a Fully Connected Neural Network Across a Cluster原评论 ↗
推荐prajit · 2015-11

“A good explanation is the One Weird Trick paper: One weird trick for parallelizing convolutional neural networks”

推荐它作为解释模型并行与数据并行训练 CNN 的清晰说明
TensorFlow: open-source library for machine intelligence原评论 ↗
推荐varelse · 2016-05

“an approach which allows an 8-GPU Big Sur with 12.5GB/s P2P bandwidth to go toe to toe with an 8-GPU DGX-1 on training AlexNet.”

推荐这个并行训练技巧,称 8 卡机器训练 AlexNet 可与 DGX-1 比肩
NVIDIA Announces the GeForce GTX 1000 Series原评论 ↗
推荐liuliu · 2022-06

“This scheme is much convoluted but very effective when training large MLP.”

称这个方案虽然绕,但训练大型 MLP 时很有效
Techniques for Training Large Neural Networks原评论 ↗
引用cr4zy · 2014-08

“nice, achieved 6x speed up on an 8 gpu box”

引用它,称在 8 GPU 机器上取得 6 倍加速
ImageNet Large Scale Visual Recognition Competition原评论 ↗
看全部 10 条 →
引用temuze · 2015-12

“One weird trick for parallelizing ConvNets: One weird trick for parallelizing convolutional neural networks”

列出这篇并行化 ConvNet 的论文,作为 2015 年的阅读项
Ask HN: Papers you read in 2015?原评论 ↗
引用nl · 2016-07

“Everyone has been waiting for that day since One Weird Trick[1].”

拿它说明大家自 One Weird Trick 起就在等这一天
Nvidia CEO Reveals New TITAN X at Stanford Deep Learning Meetup原评论 ↗
引用oneshot908 · 2018-09

“I continue to be boggled that Alex Krizhevsky's One Weird Trick never made it to TensorFlow or anywhere else:”

感叹 Krizhevsky 的 One Weird Trick 竟一直没进 TensorFlow 等框架
Building your own deep learning computer is 10x cheaper than AWS原评论 ↗
引用liuliu · 2020-02

“Looks like what it does is similar to what Alex did a few years back with One Weird Trick paper: One weird trick for parallelizing convolutional neural networks”

指出新方法类似该论文:卷积层数据并行、全连接层模型并行
Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training原评论 ↗
被提交为 story · 1 次