“Read "Training Compute-Optimal Large Language Models" paper for some mediations on the tradeoff between model size and training data volume: Training Compute-Optimal Large Language Models”
Training Compute-Optimal Large Language Models
模型大小与训练 token 数应等比例增长
研究给定算力预算下 Transformer 语言模型的最优参数量与训练 token 数。通过训练 400 多个模型,发现当时的大模型都训练不足:计算最优时模型大小每翻一倍,训练 token 也应翻一倍。据此训练的 Chinchilla 与 Gopher 算力相同,参数为 70B、数据多 4 倍,在大量下游任务上明显优于 Gopher、GPT-3 等更大的模型,在 MMLU 上平均准确率达 67.5%。
评论者把它当作训练数据量与参数量关系的标准出处,用来说明大多数大模型训练 token 不足,小模型喂更多数据可以胜过更大的模型(如 70B 胜过 175B 的 GPT-3)。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I think you should checkout this paper which discusses the relationship of performance and the ratio of training tokens to parameter count.”
“EDIT: Figure 2 page 5, and Table 3 page 8 - might be worth checking out.”
“It's amazing to me that such a big goof was missed by so many for so long.”
“Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively.”
“In the Chinchilla paper [1], the authors suggest that most big transformer models are undertrained”
看全部 20 条 →
“the insights from the recent Chinchilla paper[1] from DeepMind”
“paper indicates that most model sizes trained so far received too little token training”
“The paper shows that you can get better performance than gpt-3 with a much smaller model”
“There is quite nice literature out there, surely John has read these papers during his bootstrap period:”
“Not true. See figure 2: Training Compute-Optimal Large Language Models”
“Given that Google/DeepMind is one of the few labs actually trying to make smaller models that are just as performant (e.g. Chinchilla: Training Compute-Optimal Large Language Models ), I'm optimistic.”
“The relationship between params/tokens/compute power is something people have studied a good deal and how it affects model performance.”
“Look at Chinchilla paper[1], 70B model outperforms 175B GPT-3 model.”
“The scaling laws in it turned out to be wrong and superseded by the Chinchilla DeepMind paper: Training Compute-Optimal Large Language Models”
“Optimal number of tokens for 7B parameters is around 140B tokens[0], and meta trained it for trillion tokens.”
“For example, it was probably retrained under Chinchilla scaling laws [1], with a lot more data and maybe a somewhat smaller parameter count.”
“Well, there sort of is a linear relationship between the ammount of training data and the quality of the model [1]”
“The Chinchilla paper[0] established a simple scaling law for Large Language Models: model size and training tokens should grow at the same pace.”
“Their 70B chinchilla model significantly outperforms the 175B GPT3 model.”