“Yup that is another recent and interesting development for sure!”
Are Emergent Abilities of Large Language Models a Mirage?
涌现能力可能只是评测指标选择造成的假象
给出对涌现能力的另一种解释:在固定模型输出下,所谓涌现来自研究者对指标的选择,而非模型行为随规模的根本变化。非线性或不连续指标会制造出看似涌现的现象,线性或连续指标则给出平滑、可预测的变化。作者用简单数学模型加三组实验验证:InstructGPT/GPT-3 系列、BIG-Bench 元分析,以及在视觉任务上人为造出此前未见的伪涌现,结论是声称的涌现在换指标或改进统计后会消失,可能并非规模扩大的基本属性。
评论者把它当作对涌现论的主要反驳:指出所谓突变多来自精确匹配这类非线性指标,换成线性指标后性能随规模平滑增长,并援引它作为 NeurIPS 2023 最佳论文。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“considering the NeurIPS 2023 best paper winner is titled:”
“considering the NeurIPS 2023 best paper winner is titled:”
“good (IMO irrefutable) evidence that AI researchers have deluded themselves about the scalability of LLMs because of motivated statistical reasoning:”
“The Stanford paper's title ends with a question mark.”
“our alternative suggests that existing claims of emergent abilities are creations of the researcher’s analyses, not fundamental changes in model behavior on specific tasks with scale.”
看全部 20 条 →
“very poor word choice - Are Emergent Abilities of Large Language Models a Mirage?”
“Idk, one obvious reason could be that emergence is a mirage.”
“The paper is at Are Emergent Abilities of Large Language Models a Mirage?”
“[2304.15004] Are Emergent Abilities of Large Language Models a Mirage? - arXiv Are Emergent Abilities of Large Language Models a Mirage?”
“If you consider PCP[O(log n),O(1)] = NP, or that only O(log n) bits are required for NP, the results of this paper seems more plausible.”
“either way it doesn't really change the claim of the paper.”
“However, if one changes from nonlinear Accuracy to linear Token Edit Distance while keeping the models”
“[1] Are Emergent Abilities of Large Language Models a Mirage?: Are Emergent Abilities of Large Language Models a Mirage?”
“[2] (447 citations) Are Emergent Abilities of Large Language Models a Mirage?”
“Alternate view: Are Emergent Abilities of Large Language Models a Mirage? Are Emergent Abilities of Large Language Models a Mirage?”
“Which we probably should not, at least not the "sudden" emergence that those researchers claimed to see.”
“Ever since [0], I thought people recognized that they don't appear out of nowhere.”
“Yes, I think this is basically an instance of the "emergent abilities mirage." Are Emergent Abilities of Large Language Models a Mirage?”
“And then there goes something like this [1], where researchers failed to control for p-value”