“To anyone else here is their excellent 2016 paper linked from the above:”
Concrete Problems in AI Safety
归纳 AI 事故风险的五个可研究的具体问题
讨论机器学习系统的意外事故,即由真实世界 AI 系统设计不当导致的非预期有害行为,并列出五个相关的实际研究问题:目标函数错误引起的避免副作用和避免奖励作弊,目标函数评估成本过高引起的可扩展监督,学习过程中的不良行为引起的安全探索和分布偏移。论文回顾已有工作,提出面向前沿 AI 系统的研究方向,并探讨如何富有成效地思考前瞻性应用的安全问题。
评论者把它作为 AI 安全问题的经典入门读物,尤其是强化学习里的安全探索,并常引用它来说明相关研究者(如 Dario Amodei)很早就在思考 AI 安全。
学习路径
读这篇之前,按顺序读书架上的 5 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮你建立强化学习的目标函数问题、对抗样本和分布偏移、不确定性估计的基础,再读五类具体的 AI 安全问题。
- 1
深度强化学习是奖励黑客、副作用等问题的主要背景。
读读 DQN 如何用奖励信号从像素学习策略,理解目标函数设定如何决定行为。
- 2
- 3
- 4
对无法辨认的输入给出高置信预测,对应系统在陌生输入上自信出错的风险。
读看噪声图被高置信度分类的例子,思考模型在分布外输入上的表现。
- 5
不确定性估计是论文讨论的安全探索与分布偏移中的核心工具。
读抓 dropout 近似贝叶斯推断得到预测不确定性的思路,不必细读推导。
- 6Concrete Problems in AI Safety
把事故风险归纳为五类具体的研究问题,如副作用、奖励黑客、可扩展监督、安全探索、分布偏移。
读读引言和每类问题的定义与例子,对照前面几篇理解各个问题在技术上的来源。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“"Concrete Problems in AI Safety" is quite good, and worth reading and thinking about.”
“Concrete Problems in AI Safety (Section 6 in particular is about safe exploration)”
“None of the tweaks on machine learning they suggest do that, though.”
“Paper: Concrete Problems in AI Safety”
“According to this paper Concrete Problems in AI Safety, OpenAI (John Schulman and Paul Christiano) explicitly indicate interest in researching transparency and strongly support work on fairness.”
看全部 14 条 →
“These papers try to summarize possible research directions:”
“and this Concrete Problems in AI Safety (done while I was still at Google, but in collaboration with OpenAI people right before I joined OpenAI).”
“See eg Concrete Problems in AI Safety or [LINK] for discussions of the problem space.”
“Some overlap here with the 2017 paper from Google Brain et al: Concrete Problems in AI Safety: Concrete Problems in AI Safety.”
“Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety.” arXiv, July 25, 2016. Concrete Problems in AI Safety.”
“Here is Dario writing about AI safety in 2016 [1].”
“Ok but Dario has been thinking about AI Safety since 2016 [1], before even GPT-1.”
“Dario has been publishing papers on AI safety since 2016, before GPT-1 or even transformers existed: Concrete Problems in AI Safety”
- Concrete Problems in AI Safety ↗guiambros · 2016-06-28 · 2 分
- Concrete Problems in AI Safety ↗sonabinu · 2016-06-30 · 17 分
- Concrete Problems in AI Safety ↗peterkshultz · 2016-07-04 · 1 分
- AI Safety: problem of accidents in machine learning systems ↗blacksmythe · 2017-11-29 · 2 分
- Concrete Problems in AI Safety ↗yusufozkan · 2026-10-11 · 2 分