引用kouteiheika · 2025-06
“yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it.”
说明可以给已有模型加层再继续训练,并举这篇论文为成功先例
复制层再继续预训练的深度扩展法(DUS),得到 107 亿参数 SOLAR
论文提出 SOLAR 10.7B,一个 107 亿参数的大语言模型,并给出深度扩展(depth up-scaling,DUS)方法:先沿深度方向扩展,再继续预训练。与用混合专家的扩展方法相比,DUS 不需要对训练和推理做复杂改动,实验显示它能简单有效地从小模型得到高性能大模型。作者在此基础上还微调出指令遵循变体 SOLAR 10.7B-Instruct,超过 Mixtral-8x7B-Instruct,两者以 Apache 2.0 许可公开。
评论者把它作为“给已有模型加层再继续训练”这一做法确有人成功做过的例子,并回忆它当年是同规模里很好的模型。
“yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it.”
“duplicated transformer layers to build a 10.7B model that outperformed 30B parameter baselines.”
“Reminds me of Solar 10.7B, which was a very good model for its size ~2 year ago”