“Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator, see [1], and [4] for a presentation based on this paper.”
Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator
揭示反向模式自动微分与定界续延的联系,并结合多阶段编程提速
论文重新审视自动微分(AD),指出反向模式 AD 与定界续延(delimited continuations)紧密相关,因此只靠运算符重载、不需要额外数据结构就能实现。作者进一步把这种写法与多阶段编程(staging)结合,得到高效实现,兼具 TensorFlow 这类显式计算图框架的性能与 PyTorch 这类纯库方法的表达力。
评论者在讨论可微编程和自动微分时引用它,作为 shift/reset 路线的代表,也提到它与多阶段编程(Scala)的结合,并吐槽标题太「俏皮」。
学习路径
读这篇之前,按顺序读书架上的 2 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出候选里与核心主题(自动微分、定界续延、多阶段编程)直接相关的很少,只能从对可微编程的动机和动态计算图两个角度补一点背景。
- 1
- 2
动态计算图框架如何在运行时批处理操作,与反向传播的实现方式相关。
读抓动态计算图的构建与批处理的想法,不必细看实验。
- 3Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator
揭示反向模式自动微分与定界续延之间的联系,并用多阶段编程提速。
读抓用 shift/reset 实现反向传播的核心构造,及其与多阶段编程结合后的速度表现。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Want, Wu, Essertel, Decker, and Rompf - Demystifying differentiable programming: shift/reset the penultimate backpropagator, a title which is entirely too cute for its own good.”
“There is a paper published in 2019 from Purdue University where they show how AD can be combined with multi-stage programming in Scala.”
- Connections Between Automatic Differentiation and Delimited Continuations ↗ericjang · 2018-04-03 · 2 分
- Demystifying Differentiable Programming (2018) ↗noelwelsh · 2018-05-06 · 3 分
- Demystifying Differentiable Programming: Shift/Reset, Penultimate Backpropagator ↗adamnemecek · 2019-08-18 · 3 分
- Demystifying Differentiable Programming (2018) ↗noelwelsh · 2020-02-16 · 110 分