“a related paper that Meta published a couple of days ago that is worth looking at:”
How much do language models memorize?
估算语言模型容量约每参数 3.6 比特,并区分记忆与泛化
论文提出一种估计模型对某个数据点掌握多少信息的新方法,并用它衡量现代语言模型的容量。作者把记忆形式化地拆成两部分:非有意记忆(模型包含的关于特定数据集的信息)和泛化(模型包含的关于真实数据生成过程的信息)。完全消除泛化后可算出总记忆量,据此估计 GPT 类模型的容量约为每参数 3.6 比特。模型会一直记忆到容量被填满,随后出现 grokking,非有意记忆下降、模型开始泛化。作者训练了数百个 50 万到 15 亿参数的 Transformer,得到联系模型容量、数据规模与成员推断的扩展定律。
HN 上的人引用它说明 LLM 的记忆容量只有约每参数 3.6 到 4 比特、容量饱和后才开始泛化,并据此讨论量化损失和训练数据规模。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“This paper is highly underrated: How much do language models memorize?”
“How much do language models memorize? tells us generalization happens in LLM once their capacity for remembering things is saturated”
“FAIR is still doing interesting and useful work on LLMs for encoders [1], training dynamics [2], and multimodality [3]”
“LLMs have limited capacity to memorize, under ~4 bits per parameter[1][2], and are trained on terabytes of data.”
“LLMs usually have about 3.6 bits of data per parameter.”
看全部 7 条 →
“the actual capacity of a model to memorize is only at most ~4 bit per parameter[1] instead of full ~8 bits we assumed here”
- How much do language models memorize? ↗mhmmmmmm · 2025-06-03 · 58 分