“These are both good reads if you find yourself tending to agree with the author”
Emergent Abilities of Large Language Models
提出大模型规模增大后涌现出新能力的概念
讨论一种不可预测的现象,称为大语言模型的涌现能力:若某项能力在较小模型中不存在而在较大模型中出现,就视为涌现,因此无法仅凭小模型的表现外推得到。作者认为这种涌现意味着继续扩大规模可能进一步拓展语言模型的能力范围。
它被当作涌现一词的原始出处反复引用,用来说明大模型规模增大后会出现意料之外的新能力,同时它也是评论者争论这些能力是否真实、是否可解释的靶子。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“This paper is pretty approachable and goes over”
“For a collection of emergent properties in LLMs, I recommend this paper [2]”
“in that paper is an extraordinarily poor communication of the idea that larger models can do more than smaller models, which should be a surprise to no one.”
“but they don't provide data for fine-tuning of their very large models:”
“I'm not sure anything here is actually meaningful.”
看全部 20 条 →
“Edit distance seems like a strange way to test if the model understands arithmetic”
“Though, the paper this article is about [2] mentions that among the "emergent" tasks they tested, 92% were measuring accuracy on either exact-string-match or multiple choice, so probably just bad metrics.”
“At a very large scale new behaviors emerge[1][2].”
“This paper also touches on a pretty fascinating phenomenon - that when scaling up large language models they seem to "naturally" obtain new emergent abilities that do not exist on smaller models.”
“We at present do not know what leads to emergence in the many now-documented examples of emergent capabilities in large NNs”
“See Emergent Abilities of Large Language Models Emergent Abilities of Large Language Models”
“Wei was the lead author of the original "Emergent Abilities" paper:”
“Another paper not from Msft showing emergent task capabilities across a variety of LLMs as scale increases.”
“Yet there are emergent behaviours from these LLMs that are both surprising and not immediately understood.”
“And there is work that suggests that LLMs do some sort of phase transition to gain numeracy skills: Emergent Abilities of Large Language Models.”
“I'm talking about the ability to generalize to new tasks: Emergent Abilities of Large Language Models”
“or if the model has been trained in a way that produces novel emergent abilities not seen in other models.”
“even when they already show signs of syntactic, semantic world building capability”
“The original paper on emergent behaviors in LLMs is from Google: Emergent Abilities of Large Language Models”