“that paper assumes training data cardinality increases with the number of training generations and is strictly accumulative”
Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
累积而非替换数据时,模型坍塌可以避免
论文指出先前关于模型坍塌的研究假设新数据会替换旧数据,而更现实的设定是数据不断累积。作者在语言模型、扩散模型和变分自编码器上实验确认,替换数据会导致坍塌,而把各代合成数据与原始真实数据一并累积则可避免坍塌;在线性模型的可解析框架中,他们还证明累积数据时测试误差有与迭代次数无关的有限上界。
HN 上的人用它反驳模型坍塌被夸大的说法,指出早期结论依赖数据被替换而非累积的假设,同时也有人强调其结论依赖严格累积的前提。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Some research since seems to add nuance to its conclusions:”
“"Model collapse" is often overstated, as this paper demonstrates: Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data”
“It used an incorrect theoretical framing that assumed that data was being replaced rather than accumulated as a result of more training (see Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data which explores this).”
“We use large amounts of synthetic data for training at work and have not observed any sort of model collapse when done properly.”
“I'll assume you mean these two given by @dorolow: Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data [OTHER_PAPER]”
- Is Model Collapse Inevitable? Breaking the Curse with Real and Synthetic Data ↗PaulHoule · 2024-04-08 · 3 分
- Is Model Collapse Inevitable? ↗tosh · 2024-05-02 · 5 分
- Is Model Collapse Inevitable? Breaking the Curse of Recursion ↗colinprince · 2024-06-02 · 2 分
- Avoiding Model Collapse via Accumulating Real and Synthetic Data ↗igorkraw · 2024-07-24 · 5 分