“Another progress I find really interesting is the Outrageously Large Neural Networks”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
稀疏门控 MoE 层:容量涨千倍,计算几乎不增
论文提出稀疏门控专家混合层(MoE),由多达数千个前馈子网络构成,可训练的门控网络为每个样本选出少数专家的稀疏组合,从而真正实现条件计算。作者把最多 1370 亿参数的 MoE 以卷积方式插入堆叠的 LSTM 层之间,用于语言建模和机器翻译,在计算成本更低的情况下取得显著更好的结果,模型容量提升超过 1000 倍而效率损失很小。
评论者把它当作 MoE 的出处,用来解释 MoE 如何按 token 路由、稀疏模型的参数量不能与稠密模型直接比较,也有人认为它最终是一次有趣但失败的尝试。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.”
“He also mentions sparsely activating only the neurons that matter, which they explore in Outrageously Large Neural Networks”
“sparsely gated outrageously large neural nets”
“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer Outrageously Large Neural Networks”
“This is a sparse model. You can't just compare the parameter count against dense models like GPT-3.”
看全部 14 条 →
“not sure the most canonical paper on mixture of experts”
“A huge misconception is that MoE is an ensemble of discrete models, when it is in fact multiple FFNN modules that share an attention and embedding module.”
“[Relevant paper].”
“In 2017, it emerged from a Geoffrey Hinton / Jeff Dean / Quoc Le publication called "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer".”
“The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between other layers in a neural network.”
“From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2].”
“The paper itself is fairly popular, with several thousand citations.”
“they also copied MoEs from Google Brain/Jagellonian University Outrageously Large Neural Networks way back in 2017, too!”
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer ↗tonybeltramelli · 2017-01-24 · 3 分
- Outrageously Large Neural Networks ↗groar · 2017-01-25 · 1 分
- Outrageously Large Neural Networks ↗tomssilver · 2017-01-25 · 3 分
- Outrageously Large Neural Networks: Up to 137B Parameters ↗serialx · 2017-01-29 · 2 分
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer ↗aaronyy · 2017-01-30 · 200 分
- Outrageously Large Neural Nets: Sparsely-Gated Mixture-of-Experts Layer (2017) ↗msoad · 2019-06-08 · 65 分