推荐PaulHoule · 2023-03
“I like this feature they are working on”
喜欢他们在做的这项功能,即让 Transformer 的上下文长度超出训练长度
提出 xPos 位置编码,让 Transformer 能在训练长度之外外推
聚焦长度外推,即在短文本上训练、在更长序列上评估。论文定义“注意力分辨率”作为外推能力的指标,并提出两点设计:显式最大化注意力分辨率的相对位置编码,以及推理时使用的分块因果注意力。语言建模实验显示该模型在内插和外推设置下都表现良好。
在讨论如何扩展 LLM 上下文长度时,评论者提到微软提出的 xPos 是对旋转位置编码做缩放的方案,建议对照阅读,并指出图 1 直观展示了注意力强度。
“I like this feature they are working on”
“One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime”
“It is also worth reviewing the paper, A Length-Extrapolatable Transformer, and xPos technique which also applies scaling to rotary embeddings: A Length-Extrapolatable Transformer”
“Xpos is another paper by microsoft tackling issues with rope and you can see figure 1 on page 4 to get a visual interpretation of the sinusoidal attention strength”