“There is nothing more to this 'discovery' than some trivial abstract mathematical spaces can be represented as conditional probability structures. Is this even a discovery?”
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
小型算法数据集上,过拟合很久之后才突然出现泛化
论文提出在小型的算法生成数据集上研究神经网络的泛化,这样数据效率、记忆、泛化和学习速度都能被细致地考察。在某些设置下,网络会经历“顿悟(grokking)”:泛化性能从随机水平一路提升到完美泛化,而且这种提升可以发生在远远越过过拟合点之后。作者还发现数据集越小,泛化所需的优化量越大,并主张这类数据集是研究过参数化网络“超越记忆的泛化”这一未被充分理解问题的好场所。
评论者把它当作“延迟泛化”现象和 grokking 一词的出处,用来说明模型在记忆之后可能学到真正的规则;也有人批评其发现并不特别,并指出论文训练的是数学运算表而非有严格语法的语言。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets Grokking”
“Both of these models can generalize to numbers it have not seen.”
“Transformers are showing strong generalisation[1] and their performance continues to surprise us as they scale[2].”
“[1] Power &al. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets Grokking”
“The "grokking" paper [1] shows that it learns fundamental principles of various algorithms and isn't just predicting text.”
看全部 13 条 →
“To some limited extent, they are already good at maths:”
“See also 'grokking' [0]”
“I expect AGI training to be more like a phase transition (or a bit like grokking Grokking)”
“As far as I know the grokking phenomenon was first observed (and the name coined) in this paper, not in any blog post:”
“The word 'vacation' is not in the paper.”
“When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar:”
“That paper did not train the models on 'a language with strong consistent grammars'.”
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets ↗stygiansonic · 2022-07-02 · 2 分
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets ↗thenobsta · 2023-04-25 · 1 分
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets ↗thunderbong · 2023-05-31 · 2 分