引用roadside_picnic · 2025-07
“Implement speculative decoding for two different sized models that share a tokenizer [0]”
介绍用两个共享 tokenizer 的不同大小模型实现推测解码
小模型先起草多个 token,大模型并行验证,输出分布不变而解码快 2–3 倍。
自回归大模型解码 K 个 token 需要串行运行 K 次。论文提出推测解码:利用更高效的近似模型先给出若干 token,再用大模型并行验证,配合一种新的采样方法,在不改变输出分布的前提下同时生成多个 token。该方法无需重新训练或改动架构即可加速现成模型;在 T5-XXL 上相对标准 T5X 实现取得 2–3 倍加速,输出完全一致。
评论者把它当作推测解码的原始出处引用,用来说明草稿 token 如何被接受或拒绝(第 2.3 节)、输出分布与目标模型完全一致,以及失败时只是失去加速。
“Implement speculative decoding for two different sized models that share a tokenizer [0]”
“The paper has the details[1] in section 2.3.”
“The original paper on it clearly talks about this. [0]”
“similar idea, but the failure mode is better.”
“Speculative decoding for LLM inference was published in 2022: Fast Inference from Transformers via Speculative Decoding”
“They introduced speculative decoding of LLMs in 2022[1]”