按年被提到
20232024 · 7 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2310.02226 · cs.CL

Think before you speak: Training Language Models With Pause Tokens

三思而后言:用暂停词元训练语言模型
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan · 2023

在输入后追加可学习的暂停词元,让模型回答前多算几步

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 12 人首次 2023-10-04 · 最近 2025-10-06
12不同的人
10活跃月份
15讨论串
19被引(OpenAlex)

语言模型通常逐个生成词元,作者让模型在输出第 K+1 个词元前处理更多隐藏向量:在输入末尾追加一串可学习的 pause 词元,并延迟提取输出到最后一个 pause 词元之后。在 1B 和 130M 参数、于 C4 上预训练的 decoder-only 模型上,只有预训练和微调都使用 pause 时推理延迟才有收益;1B 模型在 9 个任务中 8 个有提升,其中 SQuAD 的 EM 提升 18%,CommonSenseQA 提升 8%,GSM8k 准确率提升 1%。

HN 为什么提它

HN 上的人把它当作「给模型更多思考计算」的例子,引用它说明即使是不含新信息的占位词元也能提升表现,并与思维链等做对照。

方法难度 3 / 5约 40 分钟经得起时间
读之前最好知道
1Transformer 与自回归语言模型
2预训练与微调(pretraining / finetuning)
3思维链(chain-of-thought)提示
4推理时计算(inference-time compute)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐simonw · 2023-10

“There's an interesting new paper about this problem: Think before you speak”

推荐这篇新论文,用暂停 token 给模型回答前多一些计算
Want to get started with LLMs? Here's what you need to know原评论 ↗
推荐spyder · 2024-03

“Pause tokens (thinking tokens) are also an interesting method to achieve that and seems to have a positive effect on performance:”

推荐 pause token 作为有趣的方法,似乎对性能有正面作用
Yi: Open Foundation Models by 01.AI原评论 ↗
引用Difwif · 2023-10

“One idea I was interested in was after reading the paper on introducing pause tokens[1]”

读了这篇引入暂停 token 的论文后,想到给 Transformer 增加额外计算
TimeGPT-1原评论 ↗
引用og_kalu · 2023-10

“You can even even implement a computation token if you wish.”

借 pause token 的思路说,甚至可以实现专门的计算 token
Ask HN: Why are current LLMs not considered AGI?原评论 ↗
引用famouswaffles · 2024-02

“even if it's just dummy tokens without any new information, it works.”

拿它说明即使只是不含新信息的哑 token 也有效
Ask HN: Do LLMs get "better" with more processing power and or time per request?原评论 ↗
引用kelseyfrog · 2024-03

“There's a recent paper which seeks to explicitly perform time-to-think using pause tokens[1].”

指出近期论文用 pause token 明确地让模型有时间思考
How Chain-of-Thought Reasoning Helps Neural Networks Compute原评论 ↗
看全部 14 条 →
引用FeepingCreature · 2024-05

“let it be pause tokens”

把 pause token 当作思维链之外增加计算的替代方案
The "Chain of Thought" Delusions原评论 ↗
引用wongarsu · 2024-06

“You could draw parallels to allowing LLMs to emit pause tokens to get more time to think (Think before you speak and similar).”

拿它类比让 LLM 输出 pause token 来争取更多思考时间
Creativity has left the chat: The price of debiasing language models原评论 ↗
引用wongarsu · 2024-07

“are directly dependent on the model recognizing whether the answer it arrived at is one it should”

指出 pause token 依赖模型判断自己得到的答案是否该采用
The A.I. Bubble is Bursting with Ed Zitron [video]原评论 ↗
引用famouswaffles · 2024-09

“So yes you can increase compute length arbitrarily by increasing tokens.”

拿它说明增加 token 可以任意延长计算,而 pause token 本身不携带信息
Ilya Sutskever's SSI Inc raises $1B原评论 ↗
引用refulgentis · 2025-03

“There's been publications with a pause token, backspace token ([OTHER_PAPER]), or a think token ([OTHER_PAPER])”

把 pause token 与 backspace token、think token 并列为已有的出版成果
Understanding R1-Zero-Like Training: A Critical Perspective原评论 ↗
引用wlib · 2025-03

“Of course, training the model to do this (use pause tokens) on purpose works too: Think before you speak”

补充说明:有意训练模型使用 pause token 同样有效
Tencent's 'Hunyuan-T1'–The First Mamba-Powered Ultra-Large Model原评论 ↗
引用layoric · 2025-05

“"Think before you speak: Training Language Models With Pause Tokens" - Think before you speak”

贴出该论文,说明用 pause token 训练能获得类似推理的收益而无需显式思考
引用ijk · 2025-10

“You can train a special token for this purpose [2].”

指出可以为此专门训练一个特殊 token,让 LLM 回答前多一些计算
Why do LLMs freak out over the seahorse emoji?原评论 ↗
被提交为 story · 1 次