按年被提到
20232023 · 6 次2026
大模型与语言模型›可解释性与表征研究·arXiv:2301.05217 · cs.LG

Progress measures for grokking via mechanistic interpretability

通过机制可解释性得到 grokking 的进度度量
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt · 2023

逆向工程小 Transformer 的模加法算法,解释 grokking 的三个阶段

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 7 人首次 2023-01-22 · 最近 2026-01-15
7不同的人
7活跃月份
10讨论串
151被引(OpenAlex)

论文主张可以通过机制可解释性(把学到的行为逆向工程成各个组件)找到连续的“进度度量”,来理解看似不连续的涌现。作者以小 Transformer 在模加法任务上出现的 grokking 为案例,完全逆向工程出它们学到的算法:用离散傅里叶变换和三角恒等式把加法转化为圆周上的旋转,并通过分析激活与权重以及在傅里叶空间中的消融来验证。据此定义的进度度量把训练分为记忆、电路形成和清理三个连续阶段,表明 grokking 并非突变,而是权重中结构化机制逐步放大、随后记忆成分被移除的结果。

HN 为什么提它

HN 上的人把它当作“Transformer 学到的是真正算法而非纯记忆”的机制性证据,并在讨论 grokking 时引用;也有人批评它只是在小而人为的数据集和模型上成立。

实证研究难度 4 / 5约 60 分钟经得起时间
读之前最好知道
1Transformer 架构
2机制可解释性(mechanistic interpretability)
3离散傅里叶变换
4模运算
5grokking 现象

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐sanxiyn · 2023-04

“If you are interested in grokking you should have a look at this:”

推荐给对 grokking 感兴趣的人,用机制层面的进度度量解释它
Grokking in Julia原评论 ↗
批评YeGoblynQueenne · 2023-05

“The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when training a small, contrived model.”

Tiny Transformer trained for addition learns bizarre addition algorithm原评论 ↗
引用cmorez · 2023-01

“Recent work[4] strongly suggests that "grokking" (a striking but not unnatural[5] form of generalization) involves networks transitioning from memorized statistics/solutions to a general solution.”

引它说明 grokking 是网络从记忆式方案过渡到通用解的过程
Do Large Language Models learn world models or just surface statistics?原评论 ↗
引用sanxiyn · 2023-04

“For example, one layer transformer eventually learns to do modular addition completely correctly.”

举例说明单层 transformer 最终能完全正确地学会模加法
We need a much more sophisticated debate about AI原评论 ↗
引用eiz · 2023-04

“For instance here's a paper I read recently that tries to explain how a very simple transformer learns how to do modular arithmetic: Progress measures for grokking via mechanistic interpretability”

举它为例,解释一个很简单的 transformer 如何学会模加法
What Are Transformer Models and How Do They Work?原评论 ↗
引用RoboTeddy · 2023-08

“the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]”

拿它说明在模加法上训练的 transformer 自己发现了离散傅里叶变换和三角恒等式
The Myth of AI Omniscience原评论 ↗
看全部 10 条 →
引用astrange · 2024-02

“We can prove that transformers can do computation beyond memorization, and that they can learn it from examples, unless you think learning an algorithm is memorization.”

拿它证明 transformer 能从样例学出超越记忆的计算,除非把学算法也算作记忆
Every model learned by gradient descent is approximately a kernel machine (2020)原评论 ↗
引用soulofmischief · 2024-10

“Nanda, et al. successfully recovered the exact mechanism through which a transformer learned to carry out modular addition. [0]”

指出 Nanda 等人成功还原了 transformer 学会模加法的确切机制
Understanding the Limitations of Mathematical Reasoning in LLMs原评论 ↗
引用soulofmischief · 2024-10

“Here is an article where a model's learned ability to carry out modular addition is recovered.”

贴出这篇文章,其中还原了模型学到的模加法能力
OpenAI builds first chip with Broadcom and TSMC, scales back foundry ambition原评论 ↗
引用soulofmischief · 2026-01

“They form probabilistic logical circuits which approximate the world they have learned through training.”

借它说明模型形成近似其所学世界的概率逻辑电路
Claude is not a senior engineer (yet)原评论 ↗