按年被提到
20242025 · 3 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2412.09764 · cs.CL

Memory Layers at Scale

Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2025-01-02 · 最近 2025-02-14
3不同的人
2活跃月份
3讨论串
3被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Memory layers use a trainable key-value lookup mechanism to add extra parameters to a model without increasing FLOPs. Conceptually, sparsely activated memory layers complement compute-heavy dense feed-forward layers, providing dedicated capacity to store and retrieve information cheaply. This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale. On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the computation budget, as well as mixture-of-expert models when matched for both compute and parameters. We find gains are especially pronounced for factual tasks. We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论kadushka · 2025-02

“The concept of a memory module, separate from a prediction engine, makes sense to me, our brains might operate like that (short vs long term memory). Many research groups have been trying to implement this idea, recent paper from Meta comes to mind: https://arxiv.org/html/2412.09764 If they believe it’s likely to scale, they will try to scale it, so we …”

LM2: Large Memory Models原评论 ↗
被提交为 story · 2 次