引用ren_engineer · 2022-12
“there's a paper that basically shows these LLMs are pretty much tapped out in terms of quality data to train on”
引它说明 LLM 的优质训练数据基本已被用尽
预测公开人类文本将在 2026 至 2032 年间被 LLM 训练用尽
论文根据当前趋势预测训练数据需求的增长,并估算公开人类文本数据的总存量。结论是若趋势持续,模型训练所用数据集大小将在 2026 到 2032 年间与公开人类文本存量相当,过度训练时会更早。作者还探讨了合成数据、跨领域迁移学习和数据效率提升作为后续进展路径。
HN 上的人用它来支持高质量人类文本数据有限、LLM 扩展会撞上数据墙的论点。
“there's a paper that basically shows these LLMs are pretty much tapped out in terms of quality data to train on”
“which predicts that high-quality text training data will be 'used up' by 2026”
“Gives some relevant papers such as”
“And the stock of human-written text is finite: Will we run out of data? Limits of LLM scaling based on human-generated data”