按年被提到
20232023 · 2 次2026
软件工程与 AI 编程›代码生成与程序合成·arXiv:2308.02828 · cs.SE

An Empirical Study of the Non-determinism of ChatGPT in Code Generation

Shuyin Ouyang, Jie M. Zhang, Mark Harman, Meng Wang · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-09-06 · 最近 2023-12-28
2不同的人
2活跃月份
2讨论串
341被引(OpenAlex)
arXiv 摘要(候选层不写导读)

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt. Non-determinism is a potential menace to scientific conclusion validity. When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses. This paper conducts an empirical study to demonstrate that non-determinism is, indeed, high, thereby underlining the need for this behavioural change. We choose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems from three code generation benchmarks (i.e., CodeContests, APPS, and HumanEval). Our results reveal high degrees of non-determinism: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for CodeContests, APPS, and HumanEval, respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature=1). These results confirm that there is, currently, a significant threat to scientific conclusion validity. In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论eatonphil · 2023-09

“LLMs don't seem to be deterministic [0, 1, 2, 3]. So no, pinning the version wouldn't be enough. [0] https://matt-rickard.com/foundational-models-are-not-enough [1] https://arxiv.org/pdf/2308.02828.pdf [2] https://www.sitation.com/non-determinism-in-ai-llm-output/ [3] https://towardsdatascience.com/the-magic-of-llms-prompt-engineering-9c3e46130131”

I replaced 50 lines of code with a single LLM prompt原评论 ↗
被提交为 story · 1 次