“You may enjoy some works on the connections between gaussian processes, neural networks, and linear(in rkhs) models.”
Deep Neural Networks as Gaussian Processes
推导无限宽深度网络与高斯过程的精确等价并做贝叶斯预测
早已知道单隐层全连接网络在参数 i.i.d. 先验且宽度趋于无穷时等价于高斯过程(GP)。论文推导了无限宽深度网络与 GP 的精确等价,给出高效计算协方差函数的流程,并在 MNIST 和 CIFAR-10 上用这些 GP 对宽深度网络做贝叶斯推断。结果显示训练后的网络准确率随宽度增加趋近对应 GP,GP 的不确定性与网络预测误差强相关,且 GP 的预测通常优于有限宽度网络;作者还把 GP 的表现与随机神经网络中信号传播的理论联系起来。
HN 上的人引用它来说明“足够宽的神经网络在极限下就是高斯过程”,在讨论神经网络、核方法和高斯过程的联系时作为参考。
学习路径
读这篇之前,按顺序读书架上的 1 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出候选里只有一篇与高斯过程和贝叶斯视角直接相关,这条路径简短。
- 1
同样把神经网络与高斯过程联系起来,并从贝叶斯角度看待不确定性。
读dropout 与深度高斯过程近似推断的对应关系。
- 2Deep Neural Networks as Gaussian Processes
推导无限宽深度网络与高斯过程的精确等价,并做贝叶斯预测。
读逐层递推协方差函数的推导,以及用其做回归的实验。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“There is similar connection between deep fully connected neural networks and Gaussian process.”
“Deep neural networks with random weights concentrate about gaussian processes, so the two may not be so far apart in practice.”
“Very wide networks turn into Gaussian processes in the limit [0].”
“I believe it's this: Deep Neural Networks as Gaussian Processes”
“It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width.”
看全部 7 条 →
“The new comparison reminds me of this old paper which compared them to Gaussian networks:”