按年被提到
20212021 · 1 次2026
软件工程与 AI 编程›代码生成与程序合成·arXiv:2105.09938 · cs.SE

Measuring Coding Challenge Competence With APPS

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, Jacob Steinhardt · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2021-05-25 · 最近 2024-03-04
2不同的人
3活跃月份
3讨论串
149被引(OpenAlex)
arXiv 摘要(候选层不写导读)

While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论nopinsight · 2023-03

“By late 2020s, it's entirely possible that a "weaker" AGI will emerge. There will be much less need to for senior developers as well, but that could be among the least of our concerns if we cannot reliably align AI with human interests by then. Date Weakly General AI is Publicly Known https://www.metaculus.com/questions/3479/date-weakly-general-ai-is-pub…”

Post-GPT Computing原评论 ↗
评论nopinsight · 2024-03

“The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words. Interestingly, no other top models have published results on this benchmark. Claude 3 Model Card: https://www-cdn.anthropi…”

Claude 3 model family原评论 ↗
被提交为 story · 1 次