“There is some papers [0] showing that the skill and agent files reduce the reasoning effectiveness in some use cases (e.g. autogenerated)”
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
上下文文件通常不提升编码智能体成功率,推理成本却增加 20% 以上
论文检验开发者常用的 AGENTS.md 这类仓库上下文文件是否真能帮助编码智能体。作者在两种互补设置下评估任务完成表现:带有 LLM 生成上下文文件的 SWE-bench 任务,以及来自含开发者提交上下文文件的仓库的新 issue 集合。结果出人意料:提供上下文文件总体上并不提高任务成功率,平均还使推理成本增加 20% 以上,对不同 LLM、不同智能体、两类上下文文件都成立。智能体能很好地遵循文件里的指令,但流行且被模型厂商推荐的仓库概览并无帮助;上下文文件适合用来说明非标准的编码规范,其他改进应先严格评估再部署。
评论者引用它说明自动生成的 AGENTS.md 和技能文件常常没用甚至有害:成功率没有普遍提升,成本却增加 20% 以上。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Similarly, there was a paper recently that suggested the same [1]”
“A recent paper (arxiv 2602.11988 Evaluating AGENTS.md) found context files increase inference cost by 20%+ with marginal performance gains;”
“ETH Zurich tested this: LLM-generated prose context = -3% performance, +20% cost.”
“which recent research[0] suggests may be counterproductive.”
“The research is:”
看全部 13 条 →
“An ETH Zurich study on 12 repos and 5,694 PRs found that loading all rules at once hurts performance by ~3% and increases cost by 20%+.”
“Another reference: Evaluating AGENTS.md. Too many times they ignore what is defined depending on the prompt and/or context.”
“Others had different findings for commonly accepted practices[2], some you may have adopted from reading documentation”
“it performed more poorly than human editted AGENTS.md? Evaluating AGENTS.md”
“I'm planning to reorganize our context repo after finding this paper (it argues that AI generated context files can stunt the performance of models):”
“Theres a study from earlier this year which suggests the opposite:”
“There are established conventions for separating the two, and research suggests that the”
- Evaluating AGENTS.md: are they helpful for coding agents? ↗mustaphah · 2026-02-16 · 2 分