引用benob · 2025-09
“Stop thinking about text being the data. There are so many other sources, even some that you can generate.”
借它说明数据不只是文本,还有很多可生成的来源
用随机生成的数据预训练模型,并给出算法复杂度视角的理论依据
论文研究用随机生成的数据来预训练模型,并从算法复杂度角度做理论论证,建立在序列模型可训练来逼近 Solomonoff 归纳的已有研究之上。实验复现了此前的结果:这样预训练的模型在多种数据集上表现出零样本上下文学习,且性能随规模提升;作者还把结果扩展到真实数据,显示这种预训练后再微调能更快收敛、泛化更好。
评论者用它说明预训练数据不一定非得是文本,可以由迭代随机计算生成,并把它与合成数据预训练的其他讨论相互参照。
“Stop thinking about text being the data. There are so many other sources, even some that you can generate.”
“Plausible - because of papers like this one: Universal pre-training by iterated random computation”
“Reminds me of "Universal pre-training by iterated random computation" Universal pre-training by iterated random computation, with bit less formal approach.”