引用x-complexity · 2025-05
“Dummy calculations aren't even needed, if you allow the LLMs to pre-compute on the given context before inference:”
指出若允许 LLM 在推理前对给定上下文预计算,根本不需要虚设的计算
让模型在问题到来前离线“思考”上下文,测试时算力需求降约 5 倍。
扩展测试时计算能帮助 LLM 解决难题,但延迟和推理成本很高。论文提出睡眠时计算:让模型在用户提问之前就离线“思考”上下文,预测可能的问题并预先算出有用的量。在作者改造的 Stateful GSM-Symbolic 和 Stateful AIME 上,达到相同准确率所需的测试时算力约减少 5 倍;继续扩大睡眠时计算还可再提高最多 13% 和 18% 的准确率。若把它摊到同一上下文的多个相关问题上(Multi-Query GSM-Symbolic),平均每个问题的成本可降低 2.5 倍;问题越可预测,效果越好,并附有一个智能体软件工程任务的案例研究。
评论者引用它说明可以让 LLM 在问题出现之前先对给定上下文预计算,以减少推理时的算力,并与 Letta 团队的智能体记忆工作联系起来。
“Dummy calculations aren't even needed, if you allow the LLMs to pre-compute on the given context before inference:”
“we also worked on MemGPT and more recently sleep-time compute”
“from what I've seen of Letta's work and e.g. their recent paper on "sleep-time compute", they are really pushing this space forward.”
“related preprint from the letta team Sleep-time Compute”