按年被提到
20172017 · 7 次2026
大模型与语言模型›混合专家与高效训练·arXiv:1701.06538 · cs.LG

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

超大规模神经网络:稀疏门控的专家混合层
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean · 2017

稀疏门控 MoE 层:容量涨千倍,计算几乎不增

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 20 人首次 2017-01-24 · 最近 2026-07-27
20不同的人
14活跃月份
22讨论串
933被引(OpenAlex)

论文提出稀疏门控专家混合层(MoE),由多达数千个前馈子网络构成,可训练的门控网络为每个样本选出少数专家的稀疏组合,从而真正实现条件计算。作者把最多 1370 亿参数的 MoE 以卷积方式插入堆叠的 LSTM 层之间,用于语言建模和机器翻译,在计算成本更低的情况下取得显著更好的结果,模型容量提升超过 1000 倍而效率损失很小。

HN 为什么提它

评论者把它当作 MoE 的出处,用来解释 MoE 如何按 token 路由、稀疏模型的参数量不能与稠密模型直接比较,也有人认为它最终是一次有趣但失败的尝试。

方法难度 3 / 5约 45 分钟经得起时间
读之前最好知道
1LSTM 与序列模型
2Softmax 与门控机制
3条件计算与稀疏激活
4分布式训练基础

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐daddyo · 2017-07

“Another progress I find really interesting is the Outrageously Large Neural Networks”

推荐这项稀疏门控 MoE 进展,觉得非常有意思
Show HN: Xcessiv – Fully managed web application for automated machine learning原评论 ↗
批评sanxiyn · 2019-11

“My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.”

Deep Learning and Its Implications for Computer Architecture and Chip Design原评论 ↗
引用shpx · 2017-08

“He also mentions sparsely activating only the neurons that matter, which they explore in Outrageously Large Neural Networks”

指出论文探索了只激活相关神经元的稀疏做法
Jeff Dean’s Lecture for YC AI [video]原评论 ↗
引用andreyk · 2018-04

“sparsely gated outrageously large neural nets”

把它作为稀疏门控超大神经网络的出处
Skynet Today – accessible and informed coverage of AI hype and panic原评论 ↗
引用andreyk · 2018-12

“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer Outrageously Large Neural Networks”

把它列为谷歌的稀疏门控 MoE 层论文
Ask HN: Are modular neural networks an interesting avenue for further research?原评论 ↗
引用ma2rten · 2020-07

“This is a sparse model. You can't just compare the parameter count against dense models like GPT-3.”

指出这是稀疏模型,参数量不能直接与 GPT-3 这类稠密模型比
GShard: Scaling giant models with conditional computation and automatic sharding原评论 ↗
看全部 14 条 →
引用bodecker · 2023-07

“not sure the most canonical paper on mixture of experts”

列为 MoE 参考之一,但不确定它是否最经典
FTC investigating ChatGPT over potential consumer harm原评论 ↗
引用RC_ITR · 2023-07

“A huge misconception is that MoE is an ensemble of discrete models, when it is in fact multiple FFNN modules that share an attention and embedding module.”

澄清 MoE 并非离散模型集成,而是共享 attention 的多个 FFNN
How to scale LLMs better with an alternative to transformers原评论 ↗
引用rishabhjain1198 · 2023-12

“[Relevant paper].”

贴出相关论文,说明门控网络如何学会挑选专家
Mistral "Mixtral" 8x7B 32k model [magnet]原评论 ↗
引用legel · 2023-12

“In 2017, it emerged from a Geoffrey Hinton / Jeff Dean / Quoc Le publication called "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer".”

追溯 MoE 的来源,称其出自 Hinton、Dean、Le 这篇论文
Mistral: Mixtral of Experts原评论 ↗
引用magicalhippo · 2024-06

“The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between other layers in a neural network.”

概括 MoE 思路:用 Softmax 选几个专家插入网络层之间
Lamini Memory Tuning: 10x Fewer Hallucinations原评论 ↗
引用magicalhippo · 2024-06

“From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2].”

说明 MoE 的专家选择可以是 Softmax 层或直接 argmax
AI Search: The Bitter-Er Lesson原评论 ↗
引用fragmede · 2025-08

“The paper itself is fairly popular, with several thousand citations.”

指出这篇论文相当有名,被引用数千次
Hand-picked selection of articles on AI fundamentals/concepts原评论 ↗
引用yorwba · 2026-07

“they also copied MoEs from Google Brain/Jagellonian University Outrageously Large Neural Networks way back in 2017, too!”

指出后来的模型早在 2017 年就抄了这篇的 MoE
Kimi-K3 Technical Report [pdf]原评论 ↗
被提交为 story · 6 次