按年被提到
20202023 · 34 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2005.14165 · cs.CL

Language Models are Few-Shot Learners

语言模型是少样本学习者
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss 等 31 人 · 2020

1750 亿参数的 GPT-3 仅凭提示中的示例就能完成多种任务

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 72 人首次 2020-05-29 · 最近 2026-11-03
72不同的人
39活跃月份
79讨论串
27,376被引(OpenAlex)

论文训练了 1750 亿参数的自回归语言模型 GPT-3,在不做任何梯度更新或微调的情况下,只通过文本提示给出任务描述和少量示例,就在翻译、问答、完形填空等多种 NLP 任务上取得强表现,有的接近微调方法。作者同时指出它在一些数据集上仍然吃力、存在训练数据污染等方法问题,并讨论了生成文本的社会影响。

HN 为什么提它

HN 上的人用它说明规模扩大后模型能从上下文示例中学习,常引用其中的算术结果、记忆化对照和人类难以分辨生成新闻的数据,来反驳「只是在背诵」的看法,也把它当作理解 LLM 的起点。

奠基之作难度 3 / 5约 150 分钟经得起时间
读之前最好知道
1Transformer 架构
2语言模型预训练与微调(fine-tuning)
3NLP 常见基准任务
4缩放定律(scaling laws)的基本概念

学习路径

读这篇之前,按顺序读书架上的 5 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出

先掌握 Transformer 与“预训练再迁移”的范式,再看稀疏注意力和缩放定律,就能理解 GPT-3 为什么把模型做到 1750 亿参数并靠提示中的示例工作。

  1. 1
    Attention Is All You Need注意力就是你所需要的一切 · 2017

    GPT-3 的解码器架构直接来自 Transformer

    读第 3 节的缩放点积注意力、多头注意力和解码器的掩码自注意力;跳过翻译实验

  2. 2
    Universal Language Model Fine-tuning for Text Classification面向文本分类的通用语言模型微调 · 2018

    先预训练语言模型再迁移到下游任务的思路源头

    读通用语言模型微调的整体流程与 discriminative fine-tuning 的动机;理解“预训练+微调”范式

  3. 3
    BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingBERT:用于语言理解的深度双向 Transformer 预训练 · 2018

    GPT-3 要对比和超越的“预训练+微调”主流范式

    读第 3 节的掩码语言模型与微调方式;对照 GPT-3 为何不再微调

  4. 4

    GPT-3 沿用了其中交替的稠密与局部带状稀疏注意力

    读稀疏注意力的分解模式(strided / fixed);了解如何降低长序列注意力开销

  5. 5
    Scaling Laws for Neural Language Models神经语言模型的缩放定律 · 2020

    GPT-3 的规模选择以缩放定律为依据

    读损失随参数量、数据量、算力的幂律关系,以及给定算力下的最优分配

  6. 6
    Language Models are Few-Shot Learners语言模型是少样本学习者 · 2020 · 就是这篇

    目标论文:在上述基础上把模型扩到 1750 亿参数

    读第 2 节 zero/one/few-shot 的定义,第 3 节各任务结果,以及局限性讨论

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐mkl · 2020-06

“If you really want to understand, skim this, and focus especially on the graphs, as they show the scaling.”

建议想真正理解就略读它,重点看展示 scaling 的图
推荐edouard-harris · 2020-08

“Check out Figure 3.13 in the GPT-3 paper, Language Models are Few-Shot Learners.”

指向论文图 3.13:人类仅 52% 能识别 GPT-3 的新闻
Are Humans Intelligent? An AI Op-Ed原评论 ↗
推荐6gvONxR4sf7o · 2020-08

“To everybody who is arguing about its capabilities without having read the paper yet, please read it.”

劝还没读论文就争论能力的人先读,论文自己写了弱点
GPT-3 has no idea what it’s talking about原评论 ↗
推荐fossuser · 2020-08

“In fact after reading these sections of the actual paper, it's hard to believe that you could have read it yourself and taken away the idea that it was memorization? Pages 22-23 in particular.”

指向第 22-23 页,反驳 GPT-3 只是记忆的说法
My AI Timelines Have Sped Up原评论 ↗
推荐fossuser · 2020-09

“Check out GPT-3’s performance on arithmetic tasks in the original paper”

推荐看原论文中 GPT-3 的算术任务表现
Nikola: How to Parlay an Ocean of Lies into a Partnership with GM原评论 ↗
推荐mountainriver · 2021-10

“Please read “Language Models are Few Shot Learners” Language Models are Few-Shot Learners it’s the paper behind gpt-3.”

推荐读这篇 GPT-3 背后的 few-shot 论文
Tesla pulled its latest ‘Full Self Driving’ beta after testers complained原评论 ↗
看全部 20 条 →
推荐spion · 2022-12

“This is why this paper was so exciting when it came out Language Models are Few-Shot Learners”

借它说明这篇论文当年为何令人兴奋
Ask HN: How does ChatGPT work?原评论 ↗
推荐cl42 · 2023-01

“ChatGPT is build on GPT-3. GPT-3 was a big NLP development. The paper has 7000+ citations: Language Models are Few-Shot Learners It was a big deal in the NLP space.”

用 7000 多次引用说明 GPT-3 是 NLP 领域的大事
Yann LeCun: ChatGPT is 'not particularly innovative'原评论 ↗
推荐cypress66 · 2023-02

“I recommend you read that paper as it answers a lot of misconceptions you may have about llms.”

推荐读它,可澄清对 LLM 的许多误解
推荐eternalban · 2023-02

“The paper for GPT-3 however is quite clear:”

称 GPT-3 论文对此讲得相当清楚
GPT Spiritual Avisor原评论 ↗
推荐kfrzcode · 2023-05

“From [0], you'll find plenty of distraction that'll edify your baseline. Check it!”

推荐它,里面有不少内容能充实基础认识
Michael J. Fox Foundation Announces Breakthrough for Parkinson's Biomarker原评论 ↗
推荐hexaga · 2023-10

“to read through the GPT3 [0] and llama2 [1] papers for more salient descriptions of the specific mechanics involved”

推荐读 GPT3 和 llama2 论文,了解具体机制
How Transformers Work原评论 ↗
推荐WhitneyLand · 2023-12

“Then after the attention paper an important one is:”

把它列为继 attention 论文之后重要的一篇
LLM Visualization原评论 ↗
推荐photon_lines · 2025-08

“read this paper for context: Language Models are Few-Shot Learners”

推荐读这篇论文了解背景
Claude Sonnet 4 now supports 1M tokens of context原评论 ↗
推荐blackbear_ · 2026-06

“The GPT3 paper is a good starting point”

推荐 GPT3 论文作为很好的入门起点
How LLMs Work原评论 ↗
批评peterthehacker · 2020-07

“This algorithm requires many examples of text on the specific topic. Probably more than most humans would require.”

OpenAI's GPT-3 may be the biggest thing since Bitcoin原评论 ↗
批评YeGoblynQueenne · 2020-07

“Overall, this sort of "testing" of arithmetic prowess simply doesn't cut the mustard.”

Why GPT-3 Matters原评论 ↗
批评YeGoblynQueenne · 2020-07

“and the absence of "spot checks" for such instances of other operations suggests they were, actually, found- but not reported, in time-honoured fashion of not reporting negative results”

Are we in an AI Overhang?原评论 ↗
批评YeGoblynQueenne · 2020-08

“And very poor testing to ensure that the solved problems don't just happen to be in the model's training dataset to begin with”

批评测试不严谨,没排除解出的题恰在训练数据里
Recent Advances in Natural Language Processing原评论 ↗
批评nutanc · 2020-08

“Are they really few shot learners? My experiments seem to suggest otherwise.”

质疑模型是否真是 few-shot learner,自己的实验不支持
GPT-3 has no idea what it’s talking about原评论 ↗