按年被提到
20252025 · 3 次2026
AI 安全、伦理与社会影响›智能体安全与滥用·arXiv:2512.09882 · cs.AI

Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

Justin W. Lin, Eliot Krzysztof Jones, Donovan Julian Jasper, Ethan Jun-shen Ho, Anna Wu, Arnold Tianyi Yang, Neil Perry, Andy Zou, Matt Fredrikson, J. Zico Kolter, Percy Liang, Dan Boneh 等 13 人 · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2025-12-11 · 最近 2026-01-06
3不同的人
2活跃月份
4讨论串
3被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment. We evaluate ten cybersecurity professionals alongside six existing AI agents and ARTEMIS, our new agent scaffold, on a large university network consisting of ~8,000 hosts across 12 subnets. ARTEMIS is a multi-agent framework featuring dynamic prompt generation, arbitrary sub-agents, and automatic vulnerability triaging. In our comparative study, ARTEMIS placed second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate and outperforming 9 of 10 human participants. While existing scaffolds such as Codex and CyAgent underperformed relative to most human participants, ARTEMIS demonstrated technical sophistication and submission quality comparable to the strongest participants. We observe that AI agents offer advantages in systematic enumeration, parallel exploitation, and cost -- certain ARTEMIS variants cost $18/hour versus $60/hour for professional penetration testers. We also identify key capability gaps: AI agents exhibit higher false-positive rates and struggle with GUI-based tasks.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)
讨论串评论tptacek · 2026-01

“It's way too early to make firm predictions here, but if you're not already in the field it's helpful to know there's been 20 years of effort at automating "pen-testing", and the specific subset of testing this project focused on (network pentesting --- as opposed to app pentesting, which targets specifically identified network applications) is already es…”

Comparing AI agents to cybersecurity professionals in real-world pen testing原评论 ↗
讨论串评论nullcathedral · 2026-01

“I work in this space. The productivity gains from LLMs are real, but not in the "replace humans" direction. Where they shine is the interpretive grunt work: "help me figure out where the auth logic is in this obfuscated blob", "make sense of this minified JS", "what's this weird binary protocol doing.", "write me a Frida script to hook these methods and …”

Comparing AI agents to cybersecurity professionals in real-world pen testing原评论 ↗
讨论串评论Sytten · 2026-01

“Bootstrap founder in that field. Fully autonomous is just not there. The winner for this "generation" will be with human in the loop / human augmentation IMO. When VC money dries out there will be a pile of autonomous ai pentest compagnies in it.”

Comparing AI agents to cybersecurity professionals in real-world pen testing原评论 ↗
被提交为 story · 4 次