“More reading on that problem if you're curious: Lost in the Middle”
Lost in the Middle: How Language Models Use Long Contexts
相关信息放在长上下文中间时,模型表现明显下降
论文在多文档问答和键值检索两项需要从输入中找出相关信息的任务上,改变相关信息在上下文中的位置来考察语言模型。结果显示,信息位于开头或结尾时表现通常最好,位于长上下文中间时显著下降,即使是专为长上下文设计的模型也是如此。作者据此更好地解释了模型如何使用输入上下文,并提出了新的评估协议。
评论者用它说明 LLM 难以利用长上下文中间的信息,因此把上下文塞满反而会降低质量,需要做上下文管理,且这个问题可能长期存在。
学习路径
读这篇之前,按顺序读书架上的 7 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先熟悉 Transformer 以及各类长上下文与位置外推方法,再看这篇如何测量模型使用长上下文时的位置偏差。
- 1
- 2
T5 的编码器-解码器结构是论文对比解码器-only 与编码器-解码器模型的背景。
读理解文本到文本的统一形式和编码器-解码器结构。
- 3
Transformer-XL 最早系统处理固定长度上下文的限制。
读抓住段级循环与相对位置编码如何扩展有效上下文。
- 4
- 5
ALiBi 通过位置偏置实现长度外推,涉及模型能否使用更长输入的问题。
读理解按距离给注意力打线性惩罚如何支持外推。
- 6
FlashAttention 是让长上下文变得可行的底层工程手段。
读抓住分块与减少显存读写的核心思路。
- 7
- 8Lost in the Middle: How Language Models Use Long Contexts
通过多文档问答和键值检索显示模型对中间位置信息利用较差。
读抓住实验设计、U 形位置曲线,以及对架构和微调影响的分析。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“it is probably one of the worst drawbacks of this technology”
“My stance is with models ignoring long contexts, we'll have this problem for a long time.”
“We also found[1] that context stuffing leads (generally) to worse quality results, as did a recent study.[2]”
“there are things like lost in the middle problems that one may have to deal with.”
“For anyone who's curious, the paper in question, entitled,”
看全部 16 条 →
“Relevant paper: Lost in the Middle: How Language Models Use Long Contexts”
“Even full quadratic attention models seem to forget or not value information given in the middle of the prompt.”
“LLMs are known to struggle with remembering stuff that isn't at the start or end of the input.”
“degraded performance on long contexts has been observed in plenty of other models [Lost in the Middle”
“I could not help but notice the Contriever curve is so much higher on y-axis Recall than the other methods (figure 11 in Lost in the Middle).”
“the issue is with the fundamentals of how LLMs actually work. Proof: Lost in the Middle”
“Does Claude under this usage paradigm not struggle with very long contexts”
“Throwing all that at agents usually leads to poor performance due to context bloat see Lost in the Middle.”
“Not sure if this is still up to date (2023), but Lost in the Middle shows that performance degrades mostly in the middle of the context.”
“From my experience this holds true to this day.”
- Lost in the Middle: How Language Models Use Long Contexts ↗gk1 · 2023-07-07 · 1 分
- Language models use long contexts ↗jojodoi · 2023-08-02 · 1 分
- Lost in the Middle: How Language Models Use Long Contexts (2023) ↗wslh · 2026-01-30 · 2 分