引用strken · 2024-12
“I was able to find To Code, or Not To Code? Exploring Impact of Code in Pre-training, but I don't have the context to know whether it's the paper you're talking about.”
找到一篇研究预训练中代码影响的论文,但不确定是不是对方说的那篇
预训练加入代码能广泛提升非代码任务表现
论文系统研究预训练数据中的代码对代码之外任务的影响。他们在 470M 到 2.8B 参数的模型上做了大量消融,评估自然语言推理、世界知识、代码基准以及 LLM 评判的胜率,结论是代码是泛化能力的关键组成,代码质量的提升对所有任务影响尤其大。与纯文本预训练相比,加入代码使自然语言推理相对提升最多 8.2%,世界知识 4.2%,生成胜率 6.6%,代码性能提升 12 倍。
HN 评论者引它作为“用代码训练能提升其他任务表现”的证据,但摘录里相关性判断很含糊。
“I was able to find To Code, or Not To Code? Exploring Impact of Code in Pre-training, but I don't have the context to know whether it's the paper you're talking about.”
“To Code, or Not To Code? Exploring Impact of Code in Pre-training + [OTHER_PAPER] ==> code training produces downstream performance improvements on other tasks”