“The most important paper to understand this issue is”
Scaling Laws for Neural Language Models
语言模型损失随参数、数据、算力按幂律下降
论文研究语言模型交叉熵损失的经验缩放规律:损失随模型规模、数据集规模和训练算力呈幂律下降,部分趋势跨越七个数量级以上,而宽度、深度等架构细节在较宽范围内影响很小。简单的方程描述了过拟合对模型和数据规模的依赖以及训练速度对模型规模的依赖,据此可确定固定算力预算的最优分配;更大的模型样本效率明显更高,算力最优的训练是用较少数据训练很大的模型并在远未收敛时停止。
HN 上的人把它当作解释为什么大模型表现可预测地提升、训练大模型为何值得的规范论文,也常拿它和后来的 Chinchilla 缩放定律比较。
学习路径
读这篇之前,按顺序读书架上的 6 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先掌握 Transformer 语言模型的结构与子词分词,再了解批量大小、数据并行和模型缩放的已有认识,就能读懂缩放定律中各个变量的含义。
- 1
- 2
论文所用的分词方式(BPE 子词)来自这里
读BPE 合并规则的算法;理解词表大小与序列长度的关系
- 3
- 4
视觉领域按深度、宽度、分辨率复合缩放的先例
读复合缩放(compound scaling)的思想;对照语言模型的缩放方式
- 5
批量大小与训练步数、数据并行关系的实证研究
读临界批量与训练步数–批量的权衡曲线;理解论文中的 S_min 与 E_min
- 6
- 7Scaling Laws for Neural Language Models
目标论文:给出损失随规模变化的幂律
读第 1 节总结的主要结论与各幂律公式;第 6 节的算力最优分配
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“a must read IMO Scaling Laws for Neural Language Models”
“the best evidence we currently have for this was published: scaling laws for natural language understanding”
“This is the canonical paper.”
“For example, look at OpenAI's latest paper on scaling Transformers”
“There's actually a paper that attempts to quantify the scaling characteristics of NLP models from early this year.”
看全部 20 条 →
“conducted their hyperparameter sweep on significantly smaller models (see: Scaling Laws for Neural Language Models)”
“is starting to become a major trend in NLP”
“Large language models are motivated by empirical scaling law.”
“My main takeaway from that paper is that a 2x increase in training cost improves performance by 5% (100x by 42%).”
“These papers have introduced so-called "scaling laws" – predictable, robust relationships between scaling parameters and loss”
“If you scaled the architecture down significantly, it would be significantly less capable[1].”
“umm...so does OpenAI. In fact this is OpenAI discovery from [1]:”
“after which their abilities quickly become impressive (read more here: Scaling Laws for Neural Language Models)”
“The ones I had in mind were the newer 'Chinchilla' scaling laws ([OTHER_PAPER]) vs. the older 'Kaplan' scaling laws”
“E.g. look at Figure 9 in [3].”
“OpenAI’s scaling law paper[0] calls the “entropy of natural language””
“I haven’t actually read this but I know roughly what it’s about”
“2020 coauthor on scaling laws Scaling Laws for Neural Language Models”
“More data, more parameters, more compute all result in a better model per”