按年被提到
20262026 · 3 次2026
软件工程与 AI 编程›编程智能体与 AI 开发·arXiv:2605.03546 · cs.SE

ProgramBench: Can Language Models Rebuild Programs From Scratch?

John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, Ofir Press · 2026
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2026-05-07 · 最近 2026-10-31
3不同的人
2活跃月份
3讨论串
0被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论adrian_b · 2026-05

“I have found it: https://news.ycombinator.com/item?id=48045174 The study paper: https://arxiv.org/abs/2605.03546 Look at Table 3, where the cheating rates of Claude Sonnet, Claude Opus and Gemini were between 20% and 36%, during the coding benchmarks.”

Through the looking glass of benchmark hacking原评论 ↗
评论westurner · 2026-10

“ScholarlyArticle: "ProgramBench: Can Language Models Rebuild Programs From Scratch?" (2026) https://arxiv.org/abs/2605.03546 Is it that agents don't do the human-in-the-loop engineering management tasks like refactoring and docs and stopping to push for branch coverage before technical debt accrues? How many tokens per insufficient solution?”

ProgramBench: Can Language Models Rebuild Programs from Scratch?原评论 ↗
被提交为 story · 1 次