推荐lhl · 2024-02
“For those interested in reading more about how it works: Mixtral of Experts”
推荐想了解 Mixtral 工作原理的人读这篇论文
Mixtral 8x7B:稀疏专家混合,每词元只激活两个专家
介绍稀疏专家混合(SMoE)语言模型 Mixtral 8x7B:架构与 Mistral 7B 相同,但每层有 8 个前馈块(专家),路由网络为每个词元在每层选两个专家处理并合并输出。每个词元可接触 47B 参数,推理时只用 13B 活跃参数;上下文长度 32k,在所有评测基准上优于或持平 Llama 2 70B 和 GPT-3.5,数学、代码和多语言上远超 Llama 2 70B,基础和指令版均以 Apache 2.0 发布。
HN 上的人把它当作了解稀疏专家混合如何工作的参考,并引用其路由分析说明专家按词元切换、并不按主题分工。
“For those interested in reading more about how it works: Mixtral of Experts”
“This seems closely related to the "Mixtral" approach of a mixture-of-experts transformer [1]”
“surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic”
“expert(s) used can change from token to token.”