按年被提到
20232023 · 6 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2305.14705 · cs.CL

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu 等 20 人 · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-07-06 · 最近 2023-09-07
2不同的人
2活跃月份
6讨论串
63被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost. Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instructiontuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (second and third scenario), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied byFLAN-MOE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论og_kalu · 2023-07

“There's nothing new about a Mixture of Experts architecture( and it's still essentially a giant matrix of weights). They just consistently well underperformed their dense counterparts so people were surprised. But this recent paper shows MoE seem to benefit a lot better with Instruct training than their dense counterparts. https://arxiv.org/abs/2305.14705”

Open Weights原评论 ↗
评论og_kalu · 2023-07

“The reason companies/researchers haven't generally touched MoE for LLMs despite how good it sounds on paper is because they've typically sucked and undeperformed their dense counterparts. assuming this is all true, Did Open ai do anything differently here or is it just scale ? I know this very recent paper shows MoE benefit far more from Instruct tuning…”

GPT-4 Architecture原评论 ↗
评论og_kalu · 2023-07

“Sparse MoE models are neither new nor secret. The only reason you haven't seen much use of them for LLMs is because they would typically well underperform their dense counterparts. Until this paper (https://arxiv.org/abs/2305.14705) indicated they apparently benefit far more from Instruct tuning than dense models, it was mostly a "good on paper" kind of …”

Llama 2 – Meta AI原评论 ↗
评论og_kalu · 2023-09

“No. MoE models are far cheaper to train and far cheaper for inference. We're not seeing massive MoE models because they've typically well underperformed their dense counterparts. Only recently has it looked like we could get equitable performance from MoE architectures. https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2308.00951 In the first pap…”

Falcon 180B Released原评论 ↗
被提交为 story · 2 次