“I really wish that when papers like this one were created, someone took the methodology and kept running with it for every model:”
NoLiMa: Long-Context Evaluation Beyond Literal Matching
NoLiMa:去掉字面匹配后,长上下文大模型的检索能力明显下滑
常用的大海捞针(NIAH)测试里,模型可借助针与草堆之间的字面匹配来简化任务。NoLiMa 设计了问题与针词汇重叠极小的针集,要求模型推断潜在关联来定位信息。对 13 个声称支持至少 128K 上下文的 LLM 评估显示:短上下文(<1K)时表现良好,但随上下文变长大幅下降;在 32K 时 11 个模型降到各自强基线的 50% 以下,GPT-4o 也从接近满分的 99.3% 降到 69.7%。作者认为原因是缺少字面匹配时注意力在长上下文中检索更困难,推理增强和 CoT 提示也难以保持性能。
HN 上的人把它当作长上下文能力被高估的重要证据,建议所有模型都用它的方法评测,并据此建议多开新对话,也有人提醒其中评测的是较旧的模型。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I post this because this information seems very important for users of LLMs, and devs implementing LLMs in their own solutions.”
“I just learned something about LLMs: start a new chat as often as possible.”
“the rest of the scores come from the original paper [0] which was published before 3.7 was available.”
“Hello Simon, sorry for asking about this tangent here, but have you seen this paper? Is it as important as it appears to be? Should this metric be on all system cards?”
“steering away from the literal matching crushes performance already at 8k+ tokens: NoLiMa, although the models in this paper are quite old (gpt-4o ish).”
看全部 7 条 →
“to avoid known NoLiMa [0] / context rot [1] crap.”
- NoLima: Long-Context Evaluation Beyond Literal Matching ↗apsec112 · 2025-02-12 · 3 分
- NoLiMa: Long-Context Evaluation Beyond Literal Matching ↗asah · 2025-02-13 · 3 分
- NoLiMa: Long-Context Evaluation Beyond Literal Matching ↗azhenley · 2025-02-14 · 1 分
- NoLiMa: Long-Context Evaluation Beyond Literal Matching ↗azhenley · 2025-02-21 · 2 分
- NoLiMa: Long-Context Evaluation Beyond Literal Matching ↗hexhowells · 2025-02-22 · 1 分
- NoLiMa: Long-Context Evaluation Beyond Literal Matching ↗azhenley · 2025-02-23 · 1 分