“This actually works better than asking the model to self evaluate by text generation, or training with RLHF.”
Discovering Latent Knowledge in Language Models Without Supervision
在激活空间中无监督找出符合逻辑一致性的“真”方向
论文提出一种只用无标注的模型内部激活回答是非问题的方法:在激活空间里寻找满足逻辑一致性的方向,例如一个陈述与其否定的真值相反。该方法不用监督信号,也不用模型输出,在 6 个模型和 10 个问答数据集上平均比零样本准确率高 4%,把对提示的敏感度降低一半,即使模型被提示生成错误答案也能保持较高准确率。结果初步表明可以区分模型知道什么与模型说了什么。
HN 上的人在谈幻觉和模型是否有事实概念时提它,说明可以通过读取模型内部激活而不是直接提问来判断模型认为什么是真的。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“A few recent favorites of mine: Discovering Latent Knowledge in Language Models Without Supervision [LINK]”
“As for the existence of a concept of factuality, I found this paper[1] very interesting.”
“Apparently you can avoid hallucination by basically reading the model’s mind instead of asking it questions.”
“Google hasn't been able to solve it in the last decade, but some people have an idea:”
“Discovering Latent Knowledge in Language Models Without Supervision shows an unsupervised approach for probing a LLM to discover things it thinks are facts”
看全部 8 条 →
“They find a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values.”
“There is a paper showing you can infer when the model is telling the truth by finding a direction in activation space”
- Discovering Latent Knowledge in Language Models Without Supervision ↗simonebrunozzi · 2022-12-10 · 1 分
- Discovering Latent Knowledge in Language Models Without Supervision ↗dayve · 2022-12-12 · 12 分