按年被提到
20242024 · 2 次2026
大模型与语言模型›量化、解码与分词·arXiv:2404.16710 · cs.CL

LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen 等 13 人 · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-04-26 · 最近 2024-05-29
2不同的人
2活跃月份
2讨论串
78被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and checkpoints at https://github.com/facebookresearch/LayerSkip.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论mazd · 2024-05

“I strongly agree! alphaXiv (https://alphaxiv.org/) is a discussion layer on top of arXiv, and it's starting to gain a lot of traction. You can replace the arXiv URL with "alphaXiv" to get to the discussion: https://arxiv.org/abs/2404.16710 → https://alphaxiv.org/abs/2404.16710 Disclaimer: I'm currently helping out with alphaXiv -- it's a fun project ou…”

Grokked Transformers Are Implicit Reasoners原评论 ↗
被提交为 story · 1 次