“Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues.”
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
用 2,294 个真实 GitHub issue 评测模型修改代码库解决问题的能力。
论文提出 SWE-bench 评测框架,包含取自 12 个流行 Python 仓库真实 issue 及对应 pull request 的 2,294 个软件工程问题:给定代码库和 issue 描述,模型需要修改代码库来解决问题。这类问题常需要跨函数、类甚至文件协调修改,并要求与执行环境交互、处理超长上下文和复杂推理。评测显示,最先进的闭源模型和作者微调的 SWE-Llama 都只能解决最简单的 issue,表现最好的 Claude 2 仅解决 1.96%。
评论者引用它说明用真实仓库的 issue 和 PR 来评测 LLM 编程能力:当年模型只能解决最简单的 issue,需要跨多文件修改时表现很差,也有人引用其数据构成(仅含 Python 仓库)。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“it uses real world software development examples from actual PRs in popular repos”
“Edit, it's about 37%, and python-only. SWE-bench”
“SWE-bench Princeton does show that models struggle significantly with real-world issues requiring changes across multiple files and functions”
“Jimenez, C. E., Yang, J., Wettig, A., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv. SWE-bench”
- SWE-Bench: Can Language Models Resolve Real-World GitHub Issues? ↗t0e · 2023-10-12 · 1 分
- SWE-BENCH: Can language models resolve real-world GitHub issues? ↗spaniard89277 · 2024-02-07 · 1 分