按年被提到
20222023 · 2 次2026
大模型与语言模型›可解释性与表征研究·arXiv:2205.10343 · cs.LG

Towards Understanding Grokking: An Effective Theory of Representation Learning

理解 grokking:表示学习的有效理论
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, Mike Williams · 2022

用有效理论与相图解释 grokking:为何泛化滞后于过拟合

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 3 人首次 2022-12-15 · 最近 2024-06-04
3不同的人
4活跃月份
4讨论串
44被引(OpenAlex)

论文研究 grokking,即模型在过拟合训练集很久之后才泛化的现象。作者提出以有效理论为基础的微观分析,以及跨超参数的相图宏观分析,发现泛化来自结构化的表示,并观察到理解、grokking、记忆和混乱四种学习相,表示学习只发生在记忆与混乱之间的“金发姑娘区”。在 transformer 上,grokking 相更靠近记忆相,因此泛化被延迟。

HN 为什么提它

评论者借它说明 grokking 只出现在训练数据量刚好够泛化的临界区间,也把它推荐为不过度简化问题的深度学习理论读物。

实证研究难度 4 / 5约 50 分钟经得起时间
读之前最好知道
1过拟合与泛化
2表示学习(representation learning)
3相图与有效理论(effective theory)基础
4Transformer 架构

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐sinenomine · 2023-01

“If someone were to be inclined to theory, I'd just recommend reading papers that don't try oversimplify the domain:”

推荐想钻研理论的人读这类不过度简化领域的论文
Ask HN: What are the foundational texts for learning about AI/ML/NN?原评论 ↗
引用sinenomine · 2022-12

“If you try to study this in depth, these models can do something the usual "datascience"-tier ones commonly cannot: Towards Understanding Grokking [LINK]”

引用它说明这些模型能做到常规“数据科学”级模型做不到的事
How Does GPT Obtain Its Ability? Tracing Emergent Abilities of Language Models原评论 ↗
引用versteegen · 2023-02

“[2] Liu &al. Towards Understanding Grokking: An Effective Theory of Representation Learning Towards Understanding Grokking”

列为参考:在玩具问题中直接学习通用规则的理解思路
Understanding the limits of large language models原评论 ↗
引用Imnimo · 2024-06

“The reason for this is that the phenomenon seems to occur at a critical zone of having just barely enough training data to make generalization possible. See Towards Understanding Grokking for details.”

引用它说明 grokking 只在训练数据刚好够泛化的临界区出现
Grokfast: Accelerated Grokking by Amplifying Slow Gradients原评论 ↗