“This is the best one I’ve seen but it has the notable caveat that it’s a relatively short 5 minute chat session:”
People cannot distinguish GPT-4 from a human in a Turing test
预注册图灵测试:GPT-4 被判为人类的比例达 54%
研究对 ELIZA、GPT-3.5 和 GPT-4 做了随机、对照、预注册的图灵测试:参与者与真人或 AI 进行 5 分钟对话,再判断对方是否为人。GPT-4 被判为人类的比例为 54%,高于 ELIZA 的 22%,但低于真人的 67%;作者称这是首个稳健的实证,表明有人工系统能通过交互式双人图灵测试,而对参与者策略与理由的分析显示,风格和社会情感因素比传统的智能观念更重要。
评论者引用它作为「GPT-4 在图灵测试中被误认为人」的研究,同时指出会话只有 5 分钟、改变规则后仍以 50% 为阈值等问题,也有人借此说图灵测试更多反映人的易受骗。
学习路径
读这篇之前,按顺序读书架上的 3 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先从 GPT-3 和关于语言模型是否真正理解的争论建立背景,再读同一作者此前的公开图灵测试,就能理解这次预注册随机对照图灵测试想回答什么。
- 1
GPT-3.5 和 GPT-4 这条模型线的起点,提供评估对象的背景。
读大规模语言模型凭少样本提示做任务的能力;只需把握整体结论。
- 2
梳理了学界对大语言模型是否真正理解的争论,是图灵测试结果所要回应的背景。
读支持与反对大语言模型能理解的两派论点,以及它们各自的依据。
- 3
- 4People cannot distinguish GPT-4 from a human in a Turing test
目标论文:用随机对照、预注册的双人图灵测试评估 ELIZA、GPT-3.5 和 GPT-4。
读实验设计和判定为人类的比例,以及参与者判断时依据的线索。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I'm complaining that the paper changed the game so it's no longer a binary choice, but continued to treat the 50% threshold as significant.”
“That earlier result was because they botched the statistics”
“humans rated other humans as human only 2/3rds of the time: People cannot distinguish GPT-4 from a human in a Turing test”
“Here’s a paper: People cannot distinguish GPT-4 from a human in a Turing test”
“Here's the link for the moment: People cannot distinguish GPT-4 from a human in a Turing test”
看全部 11 条 →
“The sys prompt given to run the turing test (from People cannot distinguish GPT-4 from a human in a Turing test) actually works well.”
“that ran a randomized study on (the old) GPT 4 with a 5 minute duration”
“That researchers haven't done Turing-test studies (taking the setup from turing and even called them that) is patently false. Including openly testing LLMs:”
“or a prompt for "People cannot distinguish GPT-4 from a human in a Turing test" People cannot distinguish GPT-4 from a human in a Turing test”
“the more evidence we get that the Turing Test about human gullibility, rather than saying anything meaningful about the machine”
- People cannot distinguish GPT-4 from a human in a Turing test ↗Paul-Craft · 2024-05-20 · 3 分
- People cannot distinguish GPT-4 from a human in a Turing test ↗PaulHoule · 2024-06-27 · 1 分