“If you really want to understand, skim this, and focus especially on the graphs, as they show the scaling.”
Language Models are Few-Shot Learners
1750 亿参数的 GPT-3 仅凭提示中的示例就能完成多种任务
论文训练了 1750 亿参数的自回归语言模型 GPT-3,在不做任何梯度更新或微调的情况下,只通过文本提示给出任务描述和少量示例,就在翻译、问答、完形填空等多种 NLP 任务上取得强表现,有的接近微调方法。作者同时指出它在一些数据集上仍然吃力、存在训练数据污染等方法问题,并讨论了生成文本的社会影响。
HN 上的人用它说明规模扩大后模型能从上下文示例中学习,常引用其中的算术结果、记忆化对照和人类难以分辨生成新闻的数据,来反驳「只是在背诵」的看法,也把它当作理解 LLM 的起点。
学习路径
读这篇之前,按顺序读书架上的 5 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先掌握 Transformer 与“预训练再迁移”的范式,再看稀疏注意力和缩放定律,就能理解 GPT-3 为什么把模型做到 1750 亿参数并靠提示中的示例工作。
- 1
- 2
先预训练语言模型再迁移到下游任务的思路源头
读通用语言模型微调的整体流程与 discriminative fine-tuning 的动机;理解“预训练+微调”范式
- 3
GPT-3 要对比和超越的“预训练+微调”主流范式
读第 3 节的掩码语言模型与微调方式;对照 GPT-3 为何不再微调
- 4
GPT-3 沿用了其中交替的稠密与局部带状稀疏注意力
读稀疏注意力的分解模式(strided / fixed);了解如何降低长序列注意力开销
- 5
- 6Language Models are Few-Shot Learners
目标论文:在上述基础上把模型扩到 1750 亿参数
读第 2 节 zero/one/few-shot 的定义,第 3 节各任务结果,以及局限性讨论
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Check out Figure 3.13 in the GPT-3 paper, Language Models are Few-Shot Learners.”
“To everybody who is arguing about its capabilities without having read the paper yet, please read it.”
“In fact after reading these sections of the actual paper, it's hard to believe that you could have read it yourself and taken away the idea that it was memorization? Pages 22-23 in particular.”
“Check out GPT-3’s performance on arithmetic tasks in the original paper”
“Please read “Language Models are Few Shot Learners” Language Models are Few-Shot Learners it’s the paper behind gpt-3.”
看全部 20 条 →
“This is why this paper was so exciting when it came out Language Models are Few-Shot Learners”
“ChatGPT is build on GPT-3. GPT-3 was a big NLP development. The paper has 7000+ citations: Language Models are Few-Shot Learners It was a big deal in the NLP space.”
“I recommend you read that paper as it answers a lot of misconceptions you may have about llms.”
“The paper for GPT-3 however is quite clear:”
“From [0], you'll find plenty of distraction that'll edify your baseline. Check it!”
“to read through the GPT3 [0] and llama2 [1] papers for more salient descriptions of the specific mechanics involved”
“Then after the attention paper an important one is:”
“read this paper for context: Language Models are Few-Shot Learners”
“The GPT3 paper is a good starting point”
“This algorithm requires many examples of text on the specific topic. Probably more than most humans would require.”
“Overall, this sort of "testing" of arithmetic prowess simply doesn't cut the mustard.”
“and the absence of "spot checks" for such instances of other operations suggests they were, actually, found- but not reported, in time-honoured fashion of not reporting negative results”
“And very poor testing to ensure that the solved problems don't just happen to be in the model's training dataset to begin with”
“Are they really few shot learners? My experiments seem to suggest otherwise.”