“I have found it: https://news.ycombinator.com/item?id=48045174 The study paper: https://arxiv.org/abs/2605.03546 Look at Table 3, where the cheating rates of Claude Sonnet, Claude Opus and Gemini were between 20% and 36%, during the coding benchmarks.”
ProgramBench: Can Language Models Rebuild Programs From Scratch?
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“ScholarlyArticle: "ProgramBench: Can Language Models Rebuild Programs From Scratch?" (2026) https://arxiv.org/abs/2605.03546 Is it that agents don't do the human-in-the-loop engineering management tasks like refactoring and docs and stopping to push for branch coverage before technical debt accrues? How many tokens per insufficient solution?”
- ProgramBench: Can Language Models Rebuild Programs from Scratch? ↗jonbaer · 2026-05-07 · 2 分