“don't miss the (formerly over-hyped) Google Research's Titans+Atlas thread, they are quite inspiring and informative on the subject”
Titans: Learning to Memorize at Test Time
给注意力配上测试时更新的神经长期记忆模块
论文提出一个神经长期记忆模块,在测试时学习记忆历史上下文,让注意力专注当前上下文的同时利用久远信息:注意力相当于短期记忆,神经记忆相当于更持久的长期记忆。据此给出 Titans 架构家族及三种记忆接入方式的变体。在语言建模、常识推理、基因组学和时间序列任务上优于 Transformer 和近期线性循环模型,并能扩展到超过 200 万的上下文长度,在大海捞针任务上准确率更高。
评论者把它当作谷歌在「测试时学习记忆、更接近持续学习」方向上的代表作,用来谈长上下文记忆、线性 RNN 谱系以及注意力的新理解。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“There's still a lot of happening to improve memory, like the latest Titans paper: Titans”
“[13] Titans”
“The paper the article is based on [1] treats”
“Something like this, using Simon Willison's llm cli, will give you a better experience than such generated articles unless your goal is explicitly to read blogs.”
“From the blog:”
看全部 9 条 →
“Have you looked at Titan and MIRAS where they use online/updating associative memory that happens to be read out via next-token prediction?”
“Per your point 4, some current hyped work is pushing hard in this direction [1, 2, 3].”
“If anything, I think that Google's Titans[1] and Hope[2] architectures are more aligned with true continual learning”
- Titans: Learning to Memorize at Test Time ↗birriel · 2025-01-13 · 115 分
- Titans: Learning to Memorize at Test Time ↗hyperluz · 2025-01-15 · 5 分
- Titans: Learning to Memorize at Test Time ↗bicepjai · 2025-01-15 · 161 分