引用knaik94 · 2023-06
“There have been recent advances with [A] position interpolation, but those methods explore fine-tuning/loras.”
提到位置插值的新进展,但这些方法依赖微调或 LoRA
线性压缩位置索引,让 RoPE 类 LLaMA 少量微调即扩到 32K 上下文
论文提出位置插值(PI),把基于 RoPE 的预训练 LLM(如 LLaMA)的上下文窗口扩展到最高 32768,只需不到 1000 步微调。做法是把输入位置索引线性缩小到原始窗口范围,而不是外推到训练长度之外,后者可能让注意力分数灾难性地变大;理论上插值的误差上界比外推小约 600 倍,在密钥检索、语言建模和长文摘要等任务上效果良好。
评论者把它当作 Meta 提出的、通过重新缩放位置索引加少量微调来延长上下文的方法;也有人指出 RoPE 缩放此前已有类似工作。
“There have been recent advances with [A] position interpolation, but those methods explore fine-tuning/loras.”
“Meta came up with it (that paper is theirs)”
“See: Extending Context Window of Large Language Models via Positional Interpolation”
“Rope - Extending Context Window of Large Language Models via Positional Interpolation”
“RoPE scaling has been a thing for a while already: Extending Context Window of Large Language Models via Positional Interpolation”