“If you want to read the source, I recommend skipping to their follow-up paper (same authors): It's Not Just Size That Matters”
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
用完形填空式提示加微调,让小模型达到接近 GPT-3 的少样本表现
GPT-3 这类上千亿参数的预训练模型有出色的少样本能力,但训练和使用耗费大量算力,碳足迹大。论文表明,参数量小几个数量级的“更绿色”的语言模型也可获得类似 GPT-3 的表现:把文本输入转换为含任务描述的完形填空问题,并配合基于梯度的优化,再利用无标注数据可获得进一步提升。作者还指出了小模型实现成功自然语言理解所需的关键因素。
HN 上的人引用它(PET 一类方法)说明小模型也能接近 GPT-3 的少样本表现;评论还提醒哪些结果才是真正的少样本、哪些用了半监督数据。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮你先理解预训练语言模型和 GPT-3 的少样本能力与资源代价,再读用小模型做到接近效果的方法。
- 1
目标方法依赖掩码语言模型,把任务改写成完形填空正是利用了这种预训练。
读掩码语言模型预训练目标。
- 2
对 BERT 预训练的改进,属于作为小模型可用的预训练模型家族。
读预训练设置的改进,以及它与 BERT 的差异。
- 3
- 4
- 5It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
用完形填空式提示加微调,使小模型达到接近 GPT-3 的少样本表现。
读把输入改写成完形填空的 pattern 与 verbalizer,以及用无标注数据的迭代做法。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“see eg this recent paper that achieves similar results with a bit more data It's Not Just Size That Matters”
“I recently found this paper[1] claiming near GPT-3 performance with only a fraction of parameters.”
“The true few-shot results are in a few places in the paper, It's Not Just Size That Matters, labeled with '- dist'.”
- It's Not Just Size That Matters:Small Language Models Are Also Few-Shot Learners ↗diegolo · 2020-09-22 · 1 分
- It's Not Just Size That Matters: Small Models with Performance Similar to GPT-3 ↗teruakohatu · 2020-09-24 · 9 分
- It's Not Just Size That Matters Small Language Models Are Also Few-Shot Learners ↗antman · 2021-03-08 · 4 分