“Neural ODE solver just announced as a Best Paper at NeurIPS 2018 conference.”
Neural Ordinary Differential Equations
用神经网络参数化隐状态的导数,构成连续深度模型
论文提出一类新的深度网络:不再指定离散的隐藏层序列,而是用神经网络参数化隐状态的导数,输出由黑盒微分方程求解器计算。这种连续深度模型的内存开销恒定,能按输入调整计算策略,并可显式地用精度换速度;作者在连续深度残差网络和连续时间隐变量模型中演示了这些性质,还构造了可用最大似然训练的连续归一化流,并给出无需访问求解器内部即可对任意 ODE 求解器做反向传播的可扩展方法。
评论者把它当作连续深度网络、把神经网络看作微分方程的代表作来引用(它拿下了 NeurIPS 2018 最佳论文),用于讨论动力系统建模和后续研究。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先看残差网络和对残差结构的分析,把残差更新理解为欧拉离散化,再补变分自编码器和归一化流,就能读懂 Neural ODE 的连续深度模型、潜变量时间序列模型和连续归一化流。
- 1
Neural ODE 的出发点是把残差更新 h + f(h) 看成欧拉法的离散化。
读残差块的形式与跳跃连接的作用;理解这一步后连续极限就顺理成章。
- 2
分析残差块的信号传播与恒等映射,帮助看清残差更新为什么接近连续动力系统。
读恒等跳连让前向和反向信号直接传播的论证;跳过千层网络实验。
- 3
- 4
NICE 是早期的归一化流,目标论文的连续归一化流是对它的连续化推广。
读可逆变换加雅可比行列式计算概率密度的思路;理解变量变换公式即可。
- 5Neural Ordinary Differential Equations
目标论文:用神经网络参数化隐状态的导数,用 ODE 求解器计算输出,并用伴随法反传。
读伴随灵敏度方法带来的常数内存,以及连续归一化流和潜 ODE 两个应用。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I have this on my desk to grind thru - interested if anyone else has looked at it”
“The paper the article is based on is here:”
“You're a few months late:P [0]”
“ODEnets and continous normalizing flows - Neural Ordinary Differential Equations”
“Maybe by modelling dynamical systems as "neural nets" as in:”
看全部 10 条 →
“Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network.”
“I'm still intrigued by the Neural Ordinary Differential Equations paper [0] and the research in that direction.”
“One way to bridge that gap would be to use Neural Ordinary Differential Equations - they draw an equivalence between vanilla (FC layer) neural nets of constant width with differential equations”
“This paper [1] models some simple (r) NN as ODEs, and uses ODE tools to train and for inference.”
- Neural Ordinary Differential Equations ↗kumaranvpl · 2018-12-13 · 3 分
- Neural Ordinary Differential Equations ↗asparagui · 2018-12-13 · 240 分