按年被提到
20232023 · 1 次2026
机器学习与深度学习›深度学习理论与泛化·arXiv:2309.02390 · cs.LG

Explaining grokking through circuit efficiency

Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, Ramana Kumar · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-09-14 · 最近 2024-11-12
2不同的人
2活跃月份
2讨论串
24被引(OpenAlex)
arXiv 摘要(候选层不写导读)

One of the most surprising puzzles in neural network generalisation is grokking: a network with perfect training accuracy but poor generalisation will, upon further training, transition to perfect generalisation. We propose that grokking occurs when the task admits a generalising solution and a memorising solution, where the generalising solution is slower to learn but more efficient, producing larger logits with the same parameter norm. We hypothesise that memorising circuits become more inefficient with larger training datasets while generalising circuits do not, suggesting there is a critical dataset size at which memorisation and generalisation are equally efficient. We make and confirm four novel predictions about grokking, providing significant evidence in favour of our explanation. Most strikingly, we demonstrate two novel and surprising behaviours: ungrokking, in which a network regresses from perfect to low test accuracy, and semi-grokking, in which a network shows delayed generalisation to partial rather than perfect test accuracy.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论naasking · 2024-11

“Not sure what's so controversial, it's well known that LLMs can trivially be viewed as operating in higher dimensional space: https://gcptips.medium.com/a-geometric-perspective-on-large-language-models-23ec7e91c764 As for generalizing to algorithms, LLMs don't yet do this as well as humans, but they do do it: https://arxiv.org/abs/2309.02390 Finally, …”

The surprising effectiveness of test-time training for abstract reasoning [pdf]原评论 ↗
被提交为 story · 1 次