“So to answer your question, yes, the evaluation dataset is spoiled.”
Evaluating Large Language Models Trained on Code
Codex 与 HumanEval:用 GitHub 代码微调 GPT,重复采样大幅提高解题率
论文介绍 Codex,一个在 GitHub 公开代码上微调的 GPT 模型,其生产版本驱动了 GitHub Copilot,并研究它写 Python 的能力。作者发布了新评测集 HumanEval,用来衡量根据 docstring 合成程序的功能正确性:Codex 解出 28.8% 的题目,GPT-3 为 0%,GPT-J 为 11.4%。每题采样 100 次可解出 70.2%,说明重复采样是产出可用解的有效策略。论文也分析了模型的局限(如长操作链与变量绑定),并讨论了安全、安保与经济方面的影响。
评论者主要用它查证 Copilot 背后 Codex 的数字:HumanEval 上 28.8%、采样 100 次后约 70%,以及它由 GPT-3 家族微调而来;也有人质疑评测集可能已被训练数据污染。
学习路径
读这篇之前,按顺序读书架上的 7 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先补 Transformer 与缩放的基础和训练语料,再看早期代码生成与程序合成工作,最后读 Codex 与 HumanEval。
- 1
- 2
- 3
对比模型 GPT-J 的训练语料,含代码相关子集
读语料构成,特别是 GitHub 等代码来源,理解代码数据从哪来
- 4
- 5
- 6
无监督的程序语言翻译,也是用语言模型处理代码的先例
读无监督预训练在代码上的用法,以及它如何评估功能正确性
- 7
同期的编程能力评测基准,对照 HumanEval 的评测设计
读题目来源与用测试用例判断正确性的做法,对比 HumanEval
- 8Evaluating Large Language Models Trained on Code
目标论文:Codex 与 HumanEval
读HumanEval 的 pass@k 指标,以及重复采样提高解题率的结果
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“So we take from this that the problems are simple and leet code style and that they have verified that the data is not in the training set by the simple nature of simply writing the code from scratch.”
“given a common one with 60 authors assumed writing leet code style programs by hand would mean they wouldn't appear in the training data (github) and didn't even bother to check.”
“In the Codex paper[1], they have two datasets that Codex got correct about 3% of the time.”
“The paper on OpenAI Codex, the stuff powering Copilot, makes it very clear in the abstract. Evaluating Large Language Models Trained on Code”
“Yes, they mention this in the Codex paper [1]:”
看全部 17 条 →
“The Codex model on which Copilot is based had about 30% accuracy on the first solution to a coding problem, but 70% when it was allowed to generate 100 solutions and choose the one that passed unit tests.”
“Even the 300M params version of Codex outperforms it [2], and the API gives you access to a 12B version.”
“in Figure 2: Evaluating Large Language Models Trained on Code; so the work above is typically”
“In formal testing, OpenAI found that Codex could generate the correct program about 30% of the time, which is pretty low:”
“The Codex paper claims 28.8% on OpenAI's novel dataset:”
“characters. After filtering, our final dataset totaled 159 GB.”
“The best results reported are 28.8% for Codex (in perpetual preprint: Evaluating Large Language Models Trained on Code)”
“as number of attempts[0]”
“2021 coauthor on Codex Evaluating Large Language Models Trained on Code”
“One thing that is interesting is that this was anticipated by the OpenAI Codex paper (which led to GitHub Copilot) all the way back in 2021:”
“like they did in the Codex paper [1]”
- Evaluating Large Language Models Trained on Code ↗aray · 2021-07-08 · 11 分
- Evaluating Large Language Models Trained on Code(GitHub Copilot) ↗prakashqwerty · 2021-07-08 · 3 分
- Evaluating Large Language Models Trained on Code (paper about GH copilot model) ↗homarp · 2021-07-08 · 4 分