按年被提到
20222023 · 8 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:2201.02177 · cs.LG

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

顿悟(Grokking):小型算法数据集上超越过拟合的泛化
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra · 2022

小型算法数据集上,过拟合很久之后才突然出现泛化

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 15 人首次 2022-02-11 · 最近 2026-06-04
15不同的人
11活跃月份
14讨论串
178被引(OpenAlex)

论文提出在小型的算法生成数据集上研究神经网络的泛化,这样数据效率、记忆、泛化和学习速度都能被细致地考察。在某些设置下,网络会经历“顿悟(grokking)”:泛化性能从随机水平一路提升到完美泛化,而且这种提升可以发生在远远越过过拟合点之后。作者还发现数据集越小,泛化所需的优化量越大,并主张这类数据集是研究过参数化网络“超越记忆的泛化”这一未被充分理解问题的好场所。

HN 为什么提它

评论者把它当作“延迟泛化”现象和 grokking 一词的出处,用来说明模型在记忆之后可能学到真正的规则;也有人批评其发现并不特别,并指出论文训练的是数学运算表而非有严格语法的语言。

实证研究难度 3 / 5约 35 分钟经得起时间
读之前最好知道
1神经网络的过拟合与泛化
2Transformer 架构
3模运算等算法任务
4优化与训练动态

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
批评mjburgess · 2024-04

“There is nothing more to this 'discovery' than some trivial abstract mathematical spaces can be represented as conditional probability structures. Is this even a discovery?”

How do machines ‘grok’ data?原评论 ↗
引用Isinlor · 2022-02

“Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets Grokking”

贴出 Grokking 论文:网络能学到小型算法函数的结构
I asked GPT-NeoX-20B a hundred arithmetic questions原评论 ↗
引用Isinlor · 2022-02

“Both of these models can generalize to numbers it have not seen.”

指出 grokking 模型能泛化到训练中没见过的数字
I asked GPT-NeoX-20B a hundred arithmetic questions原评论 ↗
引用moconnor · 2022-04

“Transformers are showing strong generalisation[1] and their performance continues to surprise us as they scale[2].”

引用它说明 Transformer 泛化能力强,在小算法数据集上记忆后仍能泛化
Socratic Models – Composing Zero-Shot Multimodal Reasoning with Language原评论 ↗
引用versteegen · 2023-02

“[1] Power &al. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets Grokking”

给出 grokking 论文作为参考:从记忆过渡到通用规则
Understanding the limits of large language models原评论 ↗
引用quotemstr · 2023-02

“The "grokking" paper [1] shows that it learns fundamental principles of various algorithms and isn't just predicting text.”

拿它说明模型学到各种算法的基本原理,不只是预测文本
Jailbreaking ChatGPT with Dan (Do Anything Now)原评论 ↗
看全部 13 条 →
引用rhn_mk1 · 2023-03

“To some limited extent, they are already good at maths:”

引用 grokking,说明模型在一定程度上已擅长数学
And yet It Understands原评论 ↗
引用whimsicalism · 2023-04

“See also 'grokking' [0]”

提示可参见 grokking:过拟合很久之后才出现的泛化
DyLoRA: Parameter Efficient Tuning of Pre-Trained Models原评论 ↗
引用sebzim4500 · 2023-05

“I expect AGI training to be more like a phase transition (or a bit like grokking Grokking)”

用 grokking 类比,预期 AGI 训练像相变一样发生
PaLM 2 Technical Report [pdf]原评论 ↗
引用woopwoop · 2023-08

“As far as I know the grokking phenomenon was first observed (and the name coined) in this paper, not in any blog post:”

指出 grokking 现象及其名称最早出自这篇论文,而非某篇博客
Do Machine Learning Models Memorize or Generalize?原评论 ↗
引用asmodeuslucifer · 2026-02

“The word 'vacation' is not in the paper.”

指出论文里没有 vacation 这个词,也没说名称由谁创造
Who discovered grokking and why is the name hard to find?原评论 ↗
引用noosphr · 2026-06

“When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar:”

引用它说明语言语法严格一致时,权重很容易被解读成语法
"They're made out of weights"原评论 ↗
引用famouswaffles · 2026-06

“That paper did not train the models on 'a language with strong consistent grammars'.”

反驳前一条:该论文训练的是数学运算表,不是语法一致的语言
"They're made out of weights"原评论 ↗
被提交为 story · 3 次