推荐adt · 2023-03
“I wanted to point you towards OpenAI's FIM 6.9B as well.”
推荐留意 OpenAI 的 FIM 6.9B 模型
把文档中段移到末尾训练,让自回归模型学会中间填充
证明只要对数据做一个简单变换,把文档中间的一段文字移到末尾,自回归语言模型就能学会填充文本,且对大比例数据做这种变换不会损害原有的从左到右生成能力(以困惑度和采样评估)。通过一系列消融实验给出变换频率、结构和填充片段选择方式等超参数的默认设置,并建议未来的自回归模型默认采用 FIM 训练。
评论者把它作为代码模型利用光标之后上下文的训练方法来引用,称 FIM 是代码生成 LLM 的常见做法,且需要重新训练模型才能获得这一能力。
“I wanted to point you towards OpenAI's FIM 6.9B as well.”
“Unfortunately, that capability will require retraining the models:”
“It was also trained using fill-in-the middle [1].”
“FiM is a common approach for code generation LLMs.”