“If you are interested in grokking you should have a look at this:”
Progress measures for grokking via mechanistic interpretability
逆向工程小 Transformer 的模加法算法,解释 grokking 的三个阶段
论文主张可以通过机制可解释性(把学到的行为逆向工程成各个组件)找到连续的“进度度量”,来理解看似不连续的涌现。作者以小 Transformer 在模加法任务上出现的 grokking 为案例,完全逆向工程出它们学到的算法:用离散傅里叶变换和三角恒等式把加法转化为圆周上的旋转,并通过分析激活与权重以及在傅里叶空间中的消融来验证。据此定义的进度度量把训练分为记忆、电路形成和清理三个连续阶段,表明 grokking 并非突变,而是权重中结构化机制逐步放大、随后记忆成分被移除的结果。
HN 上的人把它当作“Transformer 学到的是真正算法而非纯记忆”的机制性证据,并在讨论 grokking 时引用;也有人批评它只是在小而人为的数据集和模型上成立。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when training a small, contrived model.”
“Recent work[4] strongly suggests that "grokking" (a striking but not unnatural[5] form of generalization) involves networks transitioning from memorized statistics/solutions to a general solution.”
“For example, one layer transformer eventually learns to do modular addition completely correctly.”
“For instance here's a paper I read recently that tries to explain how a very simple transformer learns how to do modular arithmetic: Progress measures for grokking via mechanistic interpretability”
“the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]”
看全部 10 条 →
“We can prove that transformers can do computation beyond memorization, and that they can learn it from examples, unless you think learning an algorithm is memorization.”
“Nanda, et al. successfully recovered the exact mechanism through which a transformer learned to carry out modular addition. [0]”
“Here is an article where a model's learned ability to carry out modular addition is recovered.”
“They form probabilistic logical circuits which approximate the world they have learned through training.”