按年被提到
20222023 · 19 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2203.15556 · cs.CL

Training Compute-Optimal Large Language Models

训练计算最优的大语言模型
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland 等 22 人 · 2022

模型大小与训练 token 数应等比例增长

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 31 人首次 2022-03-30 · 最近 2026-06-16
31不同的人
17活跃月份
33讨论串
1,737被引(OpenAlex)

研究给定算力预算下 Transformer 语言模型的最优参数量与训练 token 数。通过训练 400 多个模型,发现当时的大模型都训练不足:计算最优时模型大小每翻一倍,训练 token 也应翻一倍。据此训练的 Chinchilla 与 Gopher 算力相同,参数为 70B、数据多 4 倍,在大量下游任务上明显优于 Gopher、GPT-3 等更大的模型,在 MMLU 上平均准确率达 67.5%。

HN 为什么提它

评论者把它当作训练数据量与参数量关系的标准出处,用来说明大多数大模型训练 token 不足,小模型喂更多数据可以胜过更大的模型(如 70B 胜过 175B 的 GPT-3)。

实证研究难度 3 / 5约 60 分钟有时效性
读之前最好知道
1Transformer 语言模型
2缩放定律(scaling laws)
3训练算力与 FLOPs 估算
4预训练与评测基准(MMLU)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐sbierwagen · 2022-12

“Read "Training Compute-Optimal Large Language Models" paper for some mediations on the tradeoff between model size and training data volume: Training Compute-Optimal Large Language Models”

推荐它来思考模型规模与训练数据量之间的权衡
Ask HN: What's the value ratio of GPT3's LLM vs. its training data?原评论 ↗
推荐thewataccount · 2023-04

“I think you should checkout this paper which discusses the relationship of performance and the ratio of training tokens to parameter count.”

推荐这篇论文,讨论性能与训练 token 数和参数量之比的关系
Stability AI Launches the First of Its StableLM Suite of Language Models原评论 ↗
推荐thewataccount · 2023-05

“EDIT: Figure 2 page 5, and Table 3 page 8 - might be worth checking out.”

补充推荐看第 5 页 Figure 2 和第 8 页 Table 3
SparseGPT: Language Models Can Be Accurately Pruned in One-Shot原评论 ↗
引用_hark · 2022-04

“It's amazing to me that such a big goof was missed by so many for so long.”

感叹 Chinchilla 揭示早期模型训练不足,这么大的疏漏竟被忽视这么久
Estimating PaLM's Training Cost原评论 ↗
引用timpetri · 2022-04

“Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively.”

引用 Chinchilla 给出的 500B、1T、10T 参数模型的最优训练 token 数提问
Estimating PaLM's Training Cost原评论 ↗
引用Calavar · 2022-08

“In the Chinchilla paper [1], the authors suggest that most big transformer models are undertrained”

引用 Chinchilla 论文的观点:多数大型 Transformer 训练不足
John Carmack's new AGI company, Keen Technologies, has raised a $20M round原评论 ↗
看全部 20 条 →
引用theptip · 2022-09

“the insights from the recent Chinchilla paper[1] from DeepMind”

引用 Chinchilla 的启示:更多数据加更少参数也能达到同等性能
The road to realistic full-body deepfakes原评论 ↗
引用espadrine · 2022-10

“paper indicates that most model sizes trained so far received too little token training”

引用它指出迄今多数模型规模的训练 token 都太少
GPT-3 can create both sides of an Interactive Fiction transcript原评论 ↗
引用naillo · 2022-11

“The paper shows that you can get better performance than gpt-3 with a much smaller model”

引用它说明小得多的模型也能胜过 GPT-3
Stable Diffusion 2.0原评论 ↗
引用sinenomine · 2022-12

“There is quite nice literature out there, surely John has read these papers during his bootstrap period:”

列出这些文献,反驳没人知道从哪里入手的说法
Carmack Leaves Meta原评论 ↗
引用peaslock · 2022-12

“Not true. See figure 2: Training Compute-Optimal Large Language Models”

反驳对方说法,指向 Figure 2
Copilot Internals原评论 ↗
引用minimaxir · 2023-02

“Given that Google/DeepMind is one of the few labs actually trying to make smaller models that are just as performant (e.g. Chinchilla: Training Compute-Optimal Large Language Models ), I'm optimistic.”

以 Chinchilla 为例,说明 Google/DeepMind 在做性能不减的更小模型,因此乐观
Sundar Pichai: An important next step on our AI journey原评论 ↗
引用code_runner · 2023-02

“The relationship between params/tokens/compute power is something people have studied a good deal and how it affects model performance.”

指出参数、token 和算力之间的关系及其对模型性能的影响已被大量研究
GPT in 60 Lines of NumPy原评论 ↗
引用lossolo · 2023-02

“Look at Chinchilla paper[1], 70B model outperforms 175B GPT-3 model.”

指出 Chinchilla 的 70B 模型胜过 175B 的 GPT-3
Open source solution replicates ChatGPT training process原评论 ↗
引用cubefox · 2023-02

“The scaling laws in it turned out to be wrong and superseded by the Chinchilla DeepMind paper: Training Compute-Optimal Large Language Models”

指出之前的 scaling laws 被证明有误,已被这篇 Chinchilla 论文取代
LLaMA: A foundational, 65B-parameter large language model原评论 ↗
引用vishal0123 · 2023-02

“Optimal number of tokens for 7B parameters is around 140B tokens[0], and meta trained it for trillion tokens.”

引用它:7B 参数的最优 token 数约 140B,而 Meta 训练了万亿 token
LLaMA: A foundational, 65B-parameter large language model原评论 ↗
引用DavidSJ · 2023-03

“For example, it was probably retrained under Chinchilla scaling laws [1], with a lot more data and maybe a somewhat smaller parameter count.”

推测它大概按 Chinchilla scaling laws 重训过,数据更多、参数可能略少
LLaMA-7B in Pure C++ with full Apple Silicon support原评论 ↗
引用trc001 · 2023-03

“Well, there sort of is a linear relationship between the ammount of training data and the quality of the model [1]”

引用它说明训练数据量与模型质量之间大致呈线性关系
GPT-4 performs significantly worse on coding problems not in its training data原评论 ↗
引用jacobn · 2023-03

“The Chinchilla paper[0] established a simple scaling law for Large Language Models: model size and training tokens should grow at the same pace.”

概括 Chinchilla 的 scaling law:模型规模和训练 token 应同步增长
The Prospect of an AI Winter原评论 ↗
引用crakenzak · 2023-04

“Their 70B chinchilla model significantly outperforms the 175B GPT3 model.”

引用 70B 的 Chinchilla 明显胜过 175B 的 GPT-3,说明数据质量比参数量更重要
The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT原评论 ↗