按年被提到
20212022 · 4 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2107.06499 · cs.CL

Deduplicating Training Data Makes Language Models Better

训练数据去重让语言模型更好
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini · 2021

训练集去重:模型逐字复述训练文本的频率降至约十分之一

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 6 人首次 2022-05-02 · 最近 2025-01-02发表于 Proceedings of the conference - Association for Computational Linguistics. Meeting
6不同的人
5活跃月份
6讨论串
304被引(OpenAlex)

作者发现现有语言建模数据集含大量近似重复样本和长重复子串,导致模型无提示输出中超过 1% 是对训练数据的逐字复制。他们开发了两种去重工具(例如把 C4 中一句重复超过 6 万次的 61 词句子删掉),去重后模型输出被记忆文本的频率降到十分之一,达到同样或更好的精度所需训练步数更少,还能减少训练集与测试集的重叠。

HN 为什么提它

评论者用它说明重复数据会损害训练并更容易导致记忆化(技术意义上的抄袭),并引用其中基于 MinHash 的近似去重方法和训练集对下游任务的污染问题。

实证研究难度 3 / 5约 40 分钟经得起时间
读之前最好知道
1语言模型训练与记忆化(memorization)
2近重复检测与 MinHash
3n-gram 与子串匹配
4训练集与测试集污染

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
引用joshuamorton · 2022-05

“a number of Google authored or coauthored papers still cite it, for various reasons[1][2]”

指出尽管有争议,不少 Google 论文仍引用 Stochastic Parrots
Google fires another AI researcher who reportedly challenged findings原评论 ↗
引用p1esk · 2022-08

“The Anthropic paper [1] clearly shows an example where not deduping leads to serious quality degradation.”

引用 Anthropic 论文,示例说明不去重会严重拉低质量
Chinchilla's Wild Implications原评论 ↗
引用ijk · 2022-08

“data duplication degrades the training and makes memorization (and therefore plagiarism, in the technical sense) more likely.”

说明数据重复会损害训练,并更容易导致记忆
Exploring 12M of the 2.3B images used to train Stable Diffusion原评论 ↗
引用visarga · 2022-11

“Deduplicating Training Data Makes Language Models Better”

贴出论文标题,说明训练数据去重能让语言模型更好
Ask HN: What can I personally do to fight AGAINST the Copilot class action?原评论 ↗
引用threecheese · 2024-06

“The inline referenced [Lee et al] goes into detail, and notes that one of the algorithms is based on:”

指出文中引用的 Lee 等人论文详述了去重,其算法基于 MinHash
Large language model data pipelines and Common Crawl原评论 ↗
引用YetAnotherNick · 2025-01

“e.g. see section 'Contamination of downstream tasks' of [1].”

指向论文中“下游任务污染”一节,说明去重与评测污染的关系
30% drop in O1-preview accuracy when Putnam problems are slightly variated原评论 ↗