按年被提到
20232023 · 8 次2026
大模型与语言模型›量化、解码与分词·arXiv:2302.01318 · cs.CL

Accelerating Large Language Model Decoding with Speculative Sampling

用推测采样加速大语言模型解码
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper · 2023

小模型起草、大模型并行验证,无损加速解码约 2 到 2.5 倍

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 8 人首次 2023-02-07 · 最近 2026-05-05
8不同的人
8活跃月份
10讨论串
249被引(OpenAlex)

论文提出推测采样(speculative sampling),让一次 Transformer 调用即可生成多个 token。其依据是:并行评分由更快但较弱的草稿模型生成的短续写,延迟与从大目标模型采样单个 token 相当;再配合一种改良的拒绝采样方案,在硬件数值精度范围内保持目标模型的分布不变。在 70B 的 Chinchilla 上,分布式设置下获得 2 到 2.5 倍的解码加速,且不损失样本质量、不需改动模型。

HN 为什么提它

评论者把它当作加速 LLM 推理的标准办法来引用:小模型起草、大模型验证,质量不变而速度约提升 2 到 2.5 倍。

方法难度 3 / 5约 25 分钟经得起时间
读之前最好知道
1Transformer 解码与自回归采样
2拒绝采样(rejection sampling)
3大语言模型推理延迟
4概率分布与采样

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
引用ebalit · 2023-03

“There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course.”

指出 DeepMind 有篇关于推测采样的论文,可将 LLM 解码加速约 2 倍,但思路并不全新
Facebook LLAMA is being openly distributed via torrents原评论 ↗
引用visarga · 2023-03

“Multiply that with the 2-2.5x speedup from Speculative Sampling.”

补充推测采样带来 2 到 2.5 倍的提速,可与其他优化相乘
LLaMA-7B in Pure C++ with full Apple Silicon support原评论 ↗
引用visarga · 2023-03

“Speculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model”

拿它说明用较小 LLM 在本地解码、只偶尔用大模型校验
Run 100B+ language models at home, BitTorrent‑style原评论 ↗
引用visarga · 2023-04

“You can use it as the assistant model to a large model, it's called speculative sampling.”

指出可把小模型作大模型的助手,即推测采样,约 2.5 倍加速
Stability AI Launches the First of Its StableLM Suite of Language Models原评论 ↗
引用airgapstopgap · 2023-07

“Speculative sampling [1] would still make their first claim a lie”

指出用较小草稿模型配合主模型的推测采样仍会使对方第一个说法不成立
(Databricks cofounder) How Is ChatGPT’s Behavior Changing over Time?原评论 ↗
引用espadrine · 2023-11

“I am betting on a combination of quantization and speculative sampling with a distilled smaller set of models: Accelerating Large Language Model Decoding with Speculative Sampling”

看好量化加推测采样,再配蒸馏的小模型,并用它说明推测采样是解码提速技术
Major Outage Across ChatGPT and API原评论 ↗
看全部 8 条 →
引用lhl · 2024-09

“The implementation from the DeepMind paper achieved a 2-2.5X Accelerating Large Language Model Decoding with Speculative Sampling”

指出 DeepMind 论文中的实现取得了 2 到 2.5 倍的加速
AMD Unveils Its First Small Language Model AMD-135M原评论 ↗
引用furyofantares · 2026-05

“I know in Accelerating Large Language Model Decoding with Speculative Sampling they specify a probability that you reject a token.”

指出论文里规定了拒绝某个 token 的概率
Accelerating Gemma 4: faster inference with multi-token prediction drafters原评论 ↗
被提交为 story · 2 次