批评marcfisc · 2025-05
“Also, OpenAI has proposed ways of training LLMs to trust tool outputs less than User instructions. That also doesn't work against these attacks.”
批评 OpenAI 提出的训练 LLM 少信任工具输出的做法,对这类攻击同样无效
为 LLM 定义指令优先级并训练其忽略低优先级指令,以抵御提示注入。
论文认为提示注入、越狱等攻击的一个主要根源是 LLM 常把系统提示(应用开发者的文本)与不可信用户和第三方的文本视为同等优先级。为此提出“指令层级”,明确规定不同优先级指令冲突时模型应如何表现,并给出一种数据生成方法来示范这种分层遵循行为,教模型选择性忽略低优先级指令。把它用于 GPT-3.5 后,鲁棒性大幅提升(包括训练中未见过的攻击类型),而对标准能力的损害很小。
评论者把它作为 OpenAI 防止指令被覆盖的工作引用,有人认为模型现在已经针对安全做过训练,也有人指出这种让模型少信任工具输出的做法仍挡不住提示注入类攻击。
“Also, OpenAI has proposed ways of training LLMs to trust tool outputs less than User instructions. That also doesn't work against these attacks.”
“There's been some work done recently to prevent this concept of instructional overrides though”
“I believe they are trained for security now”