“For trained wide neural networks, you can have a look at”
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
无限宽网络的训练等价于固定核上的核梯度下降(NTK)
证明梯度下降训练人工神经网络时,网络函数沿着一个新的核(神经正切核,NTK)的核梯度演化;在无限宽极限下,NTK 收敛到显式的极限核且训练中保持不变,因此可以在函数空间而非参数空间研究训练。作者证明了数据位于球面且非线性非多项式时极限 NTK 的正定性;在最小二乘回归中,网络函数遵循线性微分方程,沿 NTK 最大主成分方向收敛最快,为早停提供了理论动机。
评论者用它解释宽网络的训练与泛化:不只是高斯过程,NTK 说明无限宽网络训练时表现得像一个固定核,也为何时发生特征学习提供了背景。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The paper says more than just that they are GPs, I believe it shows that the NTK (Neural Tangent Kernel Neural Tangent Kernel) of a large class of ANN converges at initialization.”
“Infinite wide networks are a newly active field and has recently shown some promising results, theoretically [1, 2] and empirically [3].”
“The assertion is known by the community at least since 2018, if not even well before.”
“The assertion is known by the community at least since 2018, if not even well before.”
“So there is progress both in understanding generalisation and understanding when feature learning does & does not occur.”
看全部 7 条 →
“And then the NTK paper: Neural Tangent Kernel”
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks ↗headalgorithm · 2023-01-26 · 3 分