按年被提到
20222023 · 1 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:2208.02813 · cs.LG

Towards Understanding Mixture of Experts in Deep Learning

Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, Yuanzhi Li · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-08-10 · 最近 2024-06-15
2不同的人
2活跃月份
2讨论串
45被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The Mixture-of-Experts (MoE) layer, a sparsely-activated model controlled by a router, has achieved great success in deep learning. However, the understanding of such architecture remains elusive. In this paper, we formally study how the MoE layer improves the performance of neural network learning and why the mixture model will not collapse into a single model. Our empirical results suggest that the cluster structure of the underlying problem and the non-linearity of the expert are pivotal to the success of MoE. To further understand this, we consider a challenging classification problem with intrinsic cluster structures, which is hard to learn using a single expert. Yet with the MoE layer, by choosing the experts as two-layer nonlinear convolutional neural networks (CNNs), we show that the problem can be learned successfully. Furthermore, our theory shows that the router can learn the cluster-center features, which helps divide the input complex problem into simpler linear classification sub-problems that individual experts can conquer. To our knowledge, this is the first result towards formally understanding the mechanism of the MoE layer for deep learning.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论jxf · 2023-08

“It's not quite the same thing, but the "mixture of experts" model is a popular deep learning approach that displays very good performance: see e.g., https://arxiv.org/abs/2208.02813”

MetaGPT: Meta Programming for Multi-Agent Collaborative Framework原评论 ↗
评论magicalhippo · 2024-06

“From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2]. There was also a recent post[3] about a model where they used a cross-attention layer to let the expert selection be more context aware. [1]: https://arxiv.org/abs/1701.06538 [2]: https://arxiv.org/abs/2208.02813 [3]: https://news.ycombinator.com/item?id…”

AI Search: The Bitter-Er Lesson原评论 ↗