按年被提到
20262026 · 4 次2026
软件工程与 AI 编程›编程智能体与 AI 开发·arXiv:2602.07900 · cs.SE

Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, Lingxiao Jiang · 2026
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2026-06-05 · 最近 2026-10-04
3不同的人
2活跃月份
4讨论串
1被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论rsalus · 2026-06

“I was a big proponent of encoding TDD red-green-refactor methodology into my agent workflows until recently when I made the same realization after reading this study: https://arxiv.org/pdf/2602.07900 TLDR; it found test-writing volume only weakly correlates with success and that encoding test-writing principles did not move resolution rates but _did_ mat…”

Agent Skill for TDD原评论 ↗
评论zuzululu · 2026-06

“yup and I find it weird that people still remain so defensive of the Church of TDD even against empirical studies that show its limited benefits https://arxiv.org/abs/2602.07900”

评论gregwebs · 2026-10

“The value of TDD is it does ensure that tests are written and that will help with future regressions. However, I found the same problems with agents that it has with humans (usually with humans it isn't TDD but code coverage requirements). The problem is the tests are now written to satisfy a bureaucracy rather than to properly verify the code. And there…”

Agents don't need memory, they need documentation原评论 ↗
被提交为 story · 1 次