引用moyix · 2021-10
“Although our own work shows Copilot is pretty good at adding security flaws on its own:”
引用自己团队的工作:Copilot 自己就很擅长引入安全漏洞
系统测试 Copilot 补全代码的安全性,约 40% 的生成程序有漏洞
论文系统研究 GitHub Copilot 在什么条件下、多大比例上会推荐不安全的代码。作者针对 MITRE Top 25 等高风险 CWE 设计场景,从漏洞类型、提示和领域三个维度考察,共设计 89 个场景、生成 1,689 个程序,其中约 40% 存在漏洞。
HN 上的人把它当作“AI 编程助手自己就会写出带漏洞的代码”的常用出处,但也有人指出它只测了出错频率,没有衡量人们对代码有多信任。
“Although our own work shows Copilot is pretty good at adding security flaws on its own:”
“especially considering the security issues that is autosuggested by copilot.”
“It also lines up nicely with work we did last year, which looked at the security of Copilot's completions without any humans in the loop [1]”
“The Pearce, et. al paper everyone cites about security vulnerabilities didn't look at how much people trust the code, just how frequently it gives wrong answers.”