按年被提到
20242024 · 1 次2026
软件工程与 AI 编程›代码生成与程序合成·arXiv:2402.08699 · cs.SE

Unsupervised Evaluation of Code LLMs with Round-Trip Correctness

Miltiadis Allamanis, Sheena Panthaplackel, Pengcheng Yin · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-02-21 · 最近 2026-07-24
2不同的人
2活跃月份
2讨论串
5被引(OpenAlex)
arXiv 摘要(候选层不写导读)

To evaluate code large language models (LLMs), research has relied on a few small manually curated benchmarks, such as HumanEval and MBPP, which represent a narrow part of the real-world software domains. In this work, we introduce round-trip correctness (RTC) as an alternative evaluation method. RTC allows Code LLM evaluation on a broader spectrum of real-world software domains without the need for costly human curation. RTC rests on the idea that we can ask a model to make a prediction (e.g., describe some code using natural language), feed that prediction back (e.g., synthesize code from the predicted description), and check if this round-trip leads to code that is semantically equivalent to the original input. We show how to employ RTC to evaluate code synthesis and editing. We find that RTC strongly correlates with model performance on existing narrow-domain code synthesis benchmarks while allowing us to expand to a much broader set of domains and tasks which was not previously possible without costly human annotations.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论rramadass · 2026-07

“Also see; Unsupervised Evaluation of Code LLMs with Round-Trip Correctness - https://arxiv.org/abs/2402.08699 Code repository for the above - https://github.com/google-deepmind/icml2024-roundtrip-correctness RTCE: RoundTrip Code Evaluation Framework - https://www.emergentmind.com/topics/roundtripcodeeval-rtce”

Round-Trip Correctness: A New Metric for Generative AI-Based Process Modeling原评论 ↗
被提交为 story · 1 次