“There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course.”
Accelerating Large Language Model Decoding with Speculative Sampling
小模型起草、大模型并行验证,无损加速解码约 2 到 2.5 倍
论文提出推测采样(speculative sampling),让一次 Transformer 调用即可生成多个 token。其依据是:并行评分由更快但较弱的草稿模型生成的短续写,延迟与从大目标模型采样单个 token 相当;再配合一种改良的拒绝采样方案,在硬件数值精度范围内保持目标模型的分布不变。在 70B 的 Chinchilla 上,分布式设置下获得 2 到 2.5 倍的解码加速,且不损失样本质量、不需改动模型。
评论者把它当作加速 LLM 推理的标准办法来引用:小模型起草、大模型验证,质量不变而速度约提升 2 到 2.5 倍。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Multiply that with the 2-2.5x speedup from Speculative Sampling.”
“Speculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model”
“You can use it as the assistant model to a large model, it's called speculative sampling.”
“Speculative sampling [1] would still make their first claim a lie”
“I am betting on a combination of quantization and speculative sampling with a distilled smaller set of models: Accelerating Large Language Model Decoding with Speculative Sampling”
看全部 8 条 →
“The implementation from the DeepMind paper achieved a 2-2.5X Accelerating Large Language Model Decoding with Speculative Sampling”
“I know in Accelerating Large Language Model Decoding with Speculative Sampling they specify a probability that you reject a token.”
- Accelerating Large Language Model Decoding with Speculative Sampling ↗georgehill · 2023-02-07 · 1 分
- Accelerating Large Language Model Decoding with Speculative Sampling ↗Jimmc414 · 2023-09-05 · 1 分