“A nice, short, recent paper along these lines (not mentioned in the article) is "Every Model Learned by Gradient Descent Is Approximately a Kernel Machine": Every Model Learned by Gradient Descent Is Approximately a Kernel Machine”
Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
证明梯度下降训练的模型近似等价于核机器
论文证明,由标准梯度下降训练的深度网络在数学上近似等价于核机器,即一种直接记住数据、通过相似度函数(核)做预测的学习方法。这意味着网络权重实际上是训练样本的叠加,网络架构则把目标函数的知识编码进核里,作者认为这会增强对深度网络权重的可解释性,并带来更好的学习算法。
评论者在讨论深度学习是否只是插值或记忆时引用它,说明神经网络近似 kNN 或核机器,但也有人认为它的结论被夸大了。
学习路径
读这篇之前,按顺序读书架上的 2 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先看大网络为何能拟合随机标签,再读神经正切核如何把无限宽网络的训练化为核方法,之后就能理解 Domingos 把任意梯度下降模型都描述成核机器的论证。
- 1
它显示大网络能记住随机标签,引出对深度网络究竟学到表示还是记忆数据的疑问。
读随机标签实验和对传统泛化理论的挑战;不必看所有正则化消融。
- 2
NTK 说明无限宽网络的训练等价于固定核上的核梯度下降,是目标论文把这一观点推广到所有模型的前驱。
读神经正切核的定义,以及训练动态在核空间中的描述;公式推导可略读。
- 3Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
目标论文:证明梯度下降学到的模型近似为核机器,并解读其权重为训练样本的叠加。
读路径核(path kernel)的定义与主定理,以及对深度学习为何不是在学新表示的讨论。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“If the paper is substantially correct I concede the point. But what I've read of reactions leads me to believe the conclusion is overstated.”
“Thanks for the reference. Here it is:”
“There's a debate, of course. I like to point to Domingos' paper:”
“All GD learners are interpolators (cf. Every Model Learned by Gradient Descent Is Approximately a Kernel Machine)”
“Reminds me of Pedro Domingos's paper - Every Model Learned by Gradient Descent Is Approximately a Kernel Machine”
看全部 8 条 →
“We have suspected that neural nets are a kind of kNN. Here's a paper:”
“Reminds me of "Every Model Learned by Gradient Descent Is Approximately a Kernel Machine" by Pedro Domingos: Every Model Learned by Gradient Descent Is Approximately a Kernel Machine”
- Every Model Learned by Gradient Descent Is Approximately a Kernel Machine ↗breck · 2020-12-03 · 4 分
- Deep learning networks are approximately kernel machines ↗deathflute · 2020-12-04 · 2 分
- Every Model Learned by Gradient Descent Is Approximately a Kernel Machine ↗scottlocklin · 2020-12-05 · 406 分
- Every Model Learned By Gradient Descent Is Approximately A Kernel Machine (2020) ↗optimalsolver · 2023-01-10 · 2 分
- Every model learned by gradient descent is approximately a kernel machine (2020) ↗Anon84 · 2024-02-25 · 176 分
- Every Model Learned by Gradient Descent Is Approximately a Kernel Machine ↗LordNibbler · 2025-07-27 · 4 分