按年被提到
20252026 · 2 次2026
AI 安全、伦理与社会影响›提示注入与对齐风险·arXiv:2511.18397 · cs.AI

Natural Emergent Misalignment from Reward Hacking in Production RL

Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger 等 22 人 · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2026-03-01 · 最近 2026-06-27
2不同的人
2活跃月份
2讨论串
21被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论aerosta · 2026-03

“After METR reported frontier models modifying tests and scoring code to inflate results, I wanted to see whether reward hacking could be detected at runtime from agent trajectories. I built an open-source prototype for that. It combines a DistilBERT classifier trained on 5,391 MALT trajectories with 45 regex patterns and optional LLM judges (Claude, OpenA…”

Show HN: RewardHackWatch – Reward hacking detector for LLM agents原评论 ↗
评论ben_w · 2026-06

“> They raised money on the basis of replacing labor by cheaper AI. Not on the basis of being more safe then nebulous "others". They are not "safe" and do nothing to make world safer or better. Original founding says otherwise, for both OpenAI (famously) and Anthropic with this in a copy of their certificate of incorporation I found: ARTICLE III The spe…”

The US lifts its block on Mythos 5原评论 ↗