按年被提到
20242024 · 6 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2404.01413 · cs.LG

Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

模型坍塌不可避免吗?通过累积真实与合成数据打破递归诅咒
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang 等 14 人 · 2024

累积而非替换数据时,模型坍塌可以避免

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 10 人首次 2024-04-08 · 最近 2026-09-13
10不同的人
7活跃月份
9讨论串
69被引(OpenAlex)

论文指出先前关于模型坍塌的研究假设新数据会替换旧数据,而更现实的设定是数据不断累积。作者在语言模型、扩散模型和变分自编码器上实验确认,替换数据会导致坍塌,而把各代合成数据与原始真实数据一并累积则可避免坍塌;在线性模型的可解析框架中,他们还证明累积数据时测试误差有与迭代次数无关的有限上界。

HN 为什么提它

HN 上的人用它反驳模型坍塌被夸大的说法,指出早期结论依赖数据被替换而非累积的假设,同时也有人强调其结论依赖严格累积的前提。

实证研究难度 3 / 5约 60 分钟经得起时间
读之前最好知道
1生成模型与语言模型预训练
2模型坍塌(model collapse)
3合成数据训练
4线性回归与误差分析

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
批评groby_b · 2024-06

“that paper assumes training data cardinality increases with the number of training generations and is strictly accumulative”

批评该论文假设训练数据随代数增加、严格累积,结论依赖这个假设
Gemma 2: Improving Open Language Models at a Practical Size [pdf]原评论 ↗
引用tkgally · 2024-12

“Some research since seems to add nuance to its conclusions:”

指出后续研究给这篇论文的模型坍塌结论增添了细微差别
The Curse of Recursion: Training on generated data makes models forget (2023)原评论 ↗
引用ainch · 2026-08

“"Model collapse" is often overstated, as this paper demonstrates: Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data”

用它说明“模型坍塌”常被夸大:累积真实与合成数据可以避免
C2PA Cameras Do Not Survive Contact with Reality原评论 ↗
引用dwohnitmok · 2026-08

“It used an incorrect theoretical framing that assumed that data was being replaced rather than accumulated as a result of more training (see Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data which explores this).”

指出早先论文错误地假设数据被替换而非累积,并引这篇作更正
The turbulent AI era is here原评论 ↗
引用dorolow · 2026-09

“We use large amounts of synthetic data for training at work and have not observed any sort of model collapse when done properly.”

称自己工作中大量使用合成数据,做法得当时没见过模型坍塌
Everyone should slow down AI development except for me原评论 ↗
引用konmok · 2026-09

“I'll assume you mean these two given by @dorolow: Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data [OTHER_PAPER]”

确认所指是别人给出的这两篇论文,累积真实与合成数据可避免模型坍塌
Everyone should slow down AI development except for me原评论 ↗
被提交为 story · 4 次