按年被提到
20212021 · 1 次2026
大模型与语言模型›大模型评测与真实性·arXiv:2109.07958 · cs.CL

TruthfulQA: Measuring How Models Mimic Human Falsehoods

Stephanie Lin, Jacob Hilton, Owain Evans · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2021-10-02 · 最近 2026-08-20
2不同的人
2活跃月份
2讨论串
1,361被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. We crafted questions that some humans would answer falsely due to a false belief or misconception. To perform well, models must avoid generating false answers learned from imitating human texts. We tested GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. The best model was truthful on 58% of questions, while human performance was 94%. Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution. We suggest that scaling up models alone is less promising for improving truthfulness than fine-tuning using training objectives other than imitation of text from the web.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论lyaa · 2021-10

“The problem with models like GPT-3 is that they are unable to differentiate between information sources with different "trustworthiness." They learn conspiracies and wrong claims and repeat them. It's possible to feed GPT-3 prompts that encourage it to respond with conspiracies (i.e. "who really caused 9/11?") but it also randomly responds to normal prom…”

Dublin Core's dirty little secret (2010)原评论 ↗
评论root-parent · 2026-08

“>> A remaining advantage of large language models is that as they get larger, they tend to hallucinate less First time I hear that...not really true. "Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors" - https://arxiv.org/abs/2607.00447 "Calibrated Language Models Must Hallucinate" - https://arxiv.org/abs/2311.14648 "Trut…”

If this is true, the hyperscalers are toast原评论 ↗