按年被提到
20252025 · 2 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2504.13173 · cs.LG

It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2025-12-07 · 最近 2026-01-08
3不同的人
2活跃月份
3讨论串
8被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Designing efficient and effective architectural backbones has been in the core of research efforts to enhance the capability of foundation models. Inspired by the human cognitive phenomenon of attentional bias-the natural tendency to prioritize certain events or stimuli-we reconceptualize neural architectures, including Transformers, Titans, and modern linear recurrent neural networks as associative memory modules that learn a mapping of keys and values using an internal objective, referred to as attentional bias. Surprisingly, we observed that most existing sequence models leverage either (1) dot-product similarity, or (2) L2 regression objectives as their attentional bias. Going beyond these objectives, we present a set of alternative attentional bias configurations along with their effective approximations to stabilize their training procedure. We then reinterpret forgetting mechanisms in modern deep learning architectures as a form of retention regularization, providing a novel set of forget gates for sequence models. Building upon these insights, we present Miras, a general framework to design deep learning architectures based on four choices of: (i) associative memory architecture, (ii) attentional bias objective, (iii) retention gate, and (iv) memory learning algorithm. We present three novel sequence models-Moneta, Yaad, and Memora-that go beyond the power of existing linear RNNs while maintaining a fast parallelizable training process. Our experiments show different design choices in Miras yield models with varying strengths. For example, certain instances of Miras achieve exceptional performance in special tasks such as language modeling, commonsense reasoning, and recall intensive tasks, even outperforming Transformers and other modern linear recurrent models.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论okdood64 · 2025-12

“From the blog: https://arxiv.org/abs/2501.00663 https://arxiv.org/pdf/2504.13173 Is there any other company that's openly publishing their research on AI at this level? Google should get a lot of credit for this.”

Google Titans architecture, helping AI have long-term memory原评论 ↗
评论mycall · 2025-12

“Have you looked at Titan and MIRAS where they use online/updating associative memory that happens to be read out via next-token prediction? https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/ https://arxiv.org/abs/2501.00663 https://arxiv.org/pdf/2504.13173 Much research is going into these directions, but I'm more interested i…”

The dawn of a world simulator原评论 ↗
评论psb217 · 2026-01

“Per your point 4, some current hyped work is pushing hard in this direction [1, 2, 3]. The basic idea is to think of attention as a way of implementing an associative memory. Variants like SDPA or gated linear attention can then be derived as methods for optimizing this memory online such that a particular query will return a particular value. Different a…”

The Q, K, V Matrices原评论 ↗