按年被提到
20252026 · 2 次2026
软件工程与 AI 编程›编程智能体与 AI 开发·arXiv:2512.18470 · cs.SE

SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

Tue Le, Minh V. T. Thai, Dung Nguyen Manh, Huy Phan Nhat, Nghi D. Q. Bui · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2026-01-27 · 最近 2026-06-22
2不同的人
2活跃月份
2讨论串
4被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论branafter · 2026-01

“The most accessible way to start experimenting would be the Ralph loop: https://github.com/anthropics/claude-code/tree/main/plugins/ralph-wiggum You could also work backwards from this paper: https://arxiv.org/abs/2512.18470”

ChatGPT Containers can now run bash, pip/npm install packages and download files原评论 ↗
评论keheliya · 2026-06

“There are upcoming benchmarks aimed at measuring the ability to work with brownfield tasks. (Of course, benchmarks can be gamed, but they are still better than unrealistic toy tasks that earlier generations of benchmarks used. Frontier labs are yet to use them in their tech reports or marketing material, though.:-) * SWE-EVO: Benchmarking Coding Agents i…”

GLM 5.2 vs. Opus原评论 ↗