按年被提到
20232023 · 1 次2026
软件工程与 AI 编程›代码生成与程序合成·arXiv:2306.03438 · cs.LG

Large Language Models of Code Fail at Completing Code with Potential Bugs

Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, George Karypis · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-06-07 · 最近 2025-05-05
2不同的人
2活跃月份
2讨论串
2被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence of bugs in the code context for generation, which are inevitable in software development. Therefore, we introduce and study the buggy-code completion problem, inspired by the realistic scenario of real-time code suggestion where the code context contains potential bugs -- anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with synthetic bugs derived from semantics-altering operator changes (buggy-HumanEval) and one with realistic bugs derived from user submissions to coding problems (buggy-FixEval). We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs. For instance, the passing rates of CODEGEN-2B-MONO on test cases of buggy-HumanEval drop more than 50% given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains a significant gap in post-mitigation performance.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论sigtstp · 2025-05

“I feel this makes some fundamental conceptual mistakes and is just riding the LLM wave. "Semantics" is literally behavior under execution. This is syntactical analysis by a stochastic language model. I know the NLP literature uses "semantics" to talk about representations but that is an assertion which is contested [1]. Coming back to testing, this impl…”

Semantic unit testing: test code without executing it原评论 ↗
被提交为 story · 1 次