“You might want to give a read to "Scaling Data-Constrained Language Models" [1].”
Scaling Data-Constrained Language Models
数据有限时重复训练多少轮有用,及对应的扩展定律
论文研究互联网文本可能不够用时的语言模型扩展,做了大量实验,改变数据重复程度和算力预算,规模最高到 9000 亿训练 token、90 亿参数。结果显示在固定算力下,用重复数据训练最多 4 个 epoch,损失相比唯一数据几乎没有变化;但重复更多后,增加算力的价值最终衰减到零。作者提出并验证了一个计及重复 token 和多余参数价值递减的计算最优扩展定律,还试验了用代码数据扩充、去掉常用过滤器等缓解数据稀缺的办法。
HN 上的人在讨论训练数据快用完、能否多轮重复训练时引用它,常拿它说明重复数据约 4 个 epoch 内与新数据相当、之后收益递减。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“There is some good published research about doing multiple passes over the training data”
“It does seem somewhat contrary to the findings of this paper [0] that found that old data was as good as new for at least 4 epochs.”
“This paper shows utility at 44 epochs, for instance:”
“I believe the paper being referenced is “Scaling Data-Constrained Language Models”.”
“Reusing the same data loses value after a few passes: Scaling Data-Constrained Language Models”
- Scaling Data-Constrained Language Models ↗ijk · 2023-05-30 · 4 分