按年被提到
20252025 · 3 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2507.10524 · cs.CL

Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation

Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 4 人首次 2025-07-16 · 最近 2026-09-06
4不同的人
2活跃月份
4讨论串
13被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive. Existing efficiency efforts typically target either parameter sharing or adaptive computation, leaving open the question of how to attain both simultaneously. We introduce Mixture-of-Recursions (MoR), a unified framework that combines the two axes of efficiency inside a single Recursive Transformer. MoR reuses a shared stack of layers across recursion steps to achieve parameter efficiency, while lightweight routers enable adaptive token-level thinking by dynamically assigning different recursion depths to individual tokens. This allows MoR to focus quadratic attention computation only among tokens still active at a given recursion depth, further improving memory access efficiency by selectively caching only their key-value pairs. Beyond these core mechanisms, we also propose a KV sharing variant that reuses KV pairs from the first recursion, specifically designed to further decrease memory footprint. Across model scales ranging from 135M to 1.7B parameters, MoR forms a new Pareto frontier: at equal training FLOPs and smaller model sizes, it significantly lowers validation perplexity and improves few-shot accuracy, while delivering higher throughput compared with vanilla and existing recursive baselines. These gains demonstrate that MoR is an effective path towards large-model quality without incurring large-model cost.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论verdverm · 2026-09

“Internal vector recurrence is more or less what the Universal Transformer paper proposes. This recent paper does so by sharing the transformer weights in a single recurrence setup: https://arxiv.org/abs/2607.22083, and this one does so at a per-token level: https://arxiv.org/abs/2507.10524 I agree with you about not trusting Big Ai. I'm amazed that so mu…”

What is nueralese and why is it bad原评论 ↗
被提交为 story · 3 次