按年被提到
20252025 · 2 次2026
AI 安全、伦理与社会影响›AI 风险与对齐·arXiv:2502.15657 · cs.AI

Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?

Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, Marc-Antoine Rondeau, Pierre-Luc St-Charles 等 13 人 · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2025-05-09 · 最近 2025-06-03
2不同的人
2活跃月份
2讨论串
3被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The leading AI companies are increasingly focused on building generalist AI agents -- systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI training methods. Indeed, various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation. Following the precautionary principle, we see a strong need for safer, yet still useful, alternatives to the current agency-driven trajectory. Accordingly, we propose as a core building block for further advances the development of a non-agentic AI system that is trustworthy and safe by design, which we call Scientist AI. This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans. It comprises a world model that generates theories to explain data and a question-answering inference machine. Both components operate with an explicit notion of uncertainty to mitigate the risks of overconfident predictions. In light of these considerations, a Scientist AI could be used to assist human researchers in accelerating scientific progress, including in AI safety. In particular, our system can be employed as a guardrail against AI agents that might be created despite the risks involved. Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory. We hope these arguments will motivate researchers, developers, and policymakers to favor this safer path.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论esafak · 2025-05

“In a preprint titled "Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?" (https://arxiv.org/abs/2502.15657), Yoshua Bengio et al propose a safer alternative to AGI through "scientist AIs" that explain but do not have agency, which they define as: "Agents observe their environment and act in it in order to achieve goals"…”

Bengio: A Potential Path to Safer AI Development [Through Non-Agency]原评论 ↗
评论Animats · 2025-06

“This seems to be a funding proposal for "Scientist AI."[1] Start reading around page 21. They're arguing for "model-based AI", with a "world model". But they're vague about what form that "world model" takes. This is a good idea if you can do it. But people have been bashing their head against that problem for decades. That's what Cyc was all about - bui…”

Yoshua Bengio Launches LawZero: A New Nonprofit Advancing Safe-by-Design AI原评论 ↗