按年被提到
20222023 · 19 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2206.07682 · cs.CL

Emergent Abilities of Large Language Models

大语言模型的涌现能力
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto 等 16 人 · 2022

提出大模型规模增大后涌现出新能力的概念

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 32 人首次 2023-01-24 · 最近 2026-08-27
32不同的人
19活跃月份
29讨论串
2,295被引(OpenAlex)

讨论一种不可预测的现象,称为大语言模型的涌现能力:若某项能力在较小模型中不存在而在较大模型中出现,就视为涌现,因此无法仅凭小模型的表现外推得到。作者认为这种涌现意味着继续扩大规模可能进一步拓展语言模型的能力范围。

HN 为什么提它

它被当作涌现一词的原始出处反复引用,用来说明大模型规模增大后会出现意料之外的新能力,同时它也是评论者争论这些能力是否真实、是否可解释的靶子。

观点难度 2 / 5约 30 分钟有时效性
读之前最好知道
1大语言模型基础
2缩放定律(scaling laws)
3少样本提示(few-shot prompting)
4语言模型评测基准

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐dhruvdh · 2023-02

“These are both good reads if you find yourself tending to agree with the author”

推荐它,适合倾向同意作者观点的人一读
ChatGPT Is a Blurry JPEG of the Web原评论 ↗
推荐computerex · 2023-03

“This paper is pretty approachable and goes over”

推荐它,说它通俗易懂地讲了缩放规律
Hello Dolly: Democratizing the magic of ChatGPT with open models原评论 ↗
推荐BiasRegularizer · 2023-04

“For a collection of emergent properties in LLMs, I recommend this paper [2]”

推荐它作为 LLM 涌现特性的合集
Input to GPT Matters a Lot原评论 ↗
批评kerkeslager · 2023-05

“in that paper is an extraordinarily poor communication of the idea that larger models can do more than smaller models, which should be a surprise to no one.”

Cargo Cult AI原评论 ↗
批评treprinum · 2023-07

“but they don't provide data for fine-tuning of their very large models:”

批评godelski · 2024-03

“I'm not sure anything here is actually meaningful.”

质疑它对涌现的定义是否真有意义
“Emergent” abilities in LLMs actually develop gradually and predictably – study原评论 ↗
看全部 20 条 →
批评ekez · 2024-03

“Edit distance seems like a strange way to test if the model understands arithmetic”

“Emergent” abilities in LLMs actually develop gradually and predictably – study原评论 ↗
批评n2d4 · 2024-03

“Though, the paper this article is about [2] mentions that among the "emergent" tasks they tested, 92% were measuring accuracy on either exact-string-match or multiple choice, so probably just bad metrics.”

“Emergent” abilities in LLMs actually develop gradually and predictably – study原评论 ↗
引用nl · 2023-01

“At a very large scale new behaviors emerge[1][2].”

引它说明模型规模很大时会涌现新行为
Perplexity.ai Prompt Leakage原评论 ↗
引用iliane5 · 2023-01

“This paper also touches on a pretty fascinating phenomenon - that when scaling up large language models they seem to "naturally" obtain new emergent abilities that do not exist on smaller models.”

引它说明大模型规模变大后会自然获得小模型没有的涌现能力
ChatGPT Cheat Sheet原评论 ↗
引用gwern · 2023-02

“We at present do not know what leads to emergence in the many now-documented examples of emergent capabilities in large NNs”

引它说明大型神经网络已有许多涌现能力,但原因尚不清楚
It Looks Like You’re Trying to Take over the World原评论 ↗
引用stevenhuang · 2023-03

“See Emergent Abilities of Large Language Models Emergent Abilities of Large Language Models”

引它说明模型变大后会出现涌现能力
Google Releases Paper on Embodied LLM (562b parameters)原评论 ↗
引用lhl · 2023-05

“Wei was the lead author of the original "Emergent Abilities" paper:”

指出 Wei 是这篇涌现能力原始论文的第一作者
LLMs are not greater than the sum of their parts: Stanford researchers原评论 ↗
引用adamsmith143 · 2023-05

“Another paper not from Msft showing emergent task capabilities across a variety of LLMs as scale increases.”

引它说明多种 LLM 随规模增大都出现涌现的任务能力
OpenAI chief goes before US Congress to propose licenses for building AI原评论 ↗
引用celestialcheese · 2023-05

“Yet there are emergent behaviours from these LLMs that are both surprising and not immediately understood.”

引它佐证 LLM 有令人惊讶且尚未理解的涌现行为
Cargo Cult AI原评论 ↗
引用theptip · 2023-07

“And there is work that suggests that LLMs do some sort of phase transition to gain numeracy skills: Emergent Abilities of Large Language Models.”

引它说明 LLM 可能经相变获得算术能力
Llama 2 – Meta AI原评论 ↗
引用cs702 · 2023-08

“I'm talking about the ability to generalize to new tasks: Emergent Abilities of Large Language Models”

引它说明大模型无需针对任务微调就能泛化到新任务
IBM and NASA Open Source Largest Geospatial AI Foundation Model on Hugging Face原评论 ↗
引用gardnr · 2023-10

“or if the model has been trained in a way that produces novel emergent abilities not seen in other models.”

引它说明扩大规模会带来小模型没有的新能力
Pretraining on the Test Set Is All You Need原评论 ↗
引用ignoramous · 2023-11

“even when they already show signs of syntactic, semantic world building capability”

引它说明涌现能力是只在大规模下才出现的能力
Telling GPT-4 you're scared or under pressure improves performance原评论 ↗
引用lern_too_spel · 2024-02

“The original paper on emergent behaviors in LLMs is from Google: Emergent Abilities of Large Language Models”

指出 LLM 涌现行为的原始论文来自 Google
Jeff Dean: Trends in Machine Learning [video]原评论 ↗