“No, I wasn't aware of that. This is a great link, thanks! To anyone else here is their excellent 2016 paper linked from the above: "Concrete Problems in AI Safety" https://arxiv.org/abs/1606.06565 and direct link to 29-page PDF: https://arxiv.org/pdf/1606.06565.pdf [PDF!] However, that PDF does not mention "AGI" even a single time, except in referen…”
The AGI Containment Problem
There is considerable uncertainty about what properties, capabilities and motivations future AGIs will have. In some plausible scenarios, AGIs may pose security risks arising from accidents and defects. In order to mitigate these risks, prudent early AGI research teams will perform significant testing on their creations before use. Unfortunately, if an AGI has human-level or greater intelligence, testing itself may not be safe; some natural AGI goal systems create emergent incentives for AGIs to tamper with their test environments, make copies of themselves on the internet, or convince developers and operators to do dangerous things. In this paper, we survey the AGI containment problem - the question of how to build a container in which tests can be conducted safely and reliably, even on AGIs with unknown motivations and capabilities that could be dangerous. We identify requirements for AGI containers, available mechanisms, and weaknesses that need to be addressed.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I'm the author of a previous paper about the AI containment problem. This new paper, Alfonseca et al, is kind of crap. This paper defines the term "containment problem" in a nonstandard way. This is different from how I use it in https://arxiv.org/pdf/1604.00545.pdf (which is largely about preventing side-channels and security vulnerabilities in realisti…”