按年被提到
20182018 · 1 次2026
安全、隐私与密码学›调试、模糊测试与供应链·arXiv:1808.09700 · cs.CR

Evaluating Fuzz Testing

George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, Michael Hicks · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-09-05 · 最近 2024-07-01
2不同的人
2活跃月份
2讨论串
664被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Fuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated experimentally so an important question is: What experimental setup is needed to produce trustworthy results? We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer. Our results showed that the general problems we found in existing experimental evaluations can indeed translate to actual wrong or misleading assessments. We conclude with some guidelines that we hope will help improve experimental evaluations of fuzz testing algorithms, making reported results more robust.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)
讨论串评论fulafel · 2018-09

“Choice bit from abstract: "We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer."”

Evaluating Fuzz Testing原评论 ↗
讨论串评论tyoma · 2018-09

“This paper is absolutely fascinating. Fuzzers have some inherent nondeterminism, meaning that some runs agains the same program will differ in how many bugs they find. It turns out pretty much every fuzzer evaluation to date has not been statistically rigorous. We dont actually know if a lot of published results are, well, really an improvement. There …”

Evaluating Fuzz Testing原评论 ↗
讨论串评论mwhicks1 · 2018-09

“I'm one of the authors of the paper. Two quick things I want to say: I've written a blog post that is a short but rigorous description of the paper's results: http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical-evaluations-for-fuzz-testing/ The paper does not make strong statements of the form, "fuzzers are/do X" or "all prior papers' claims ar…”

Evaluating Fuzz Testing原评论 ↗
被提交为 story · 2 次