按年被提到
20182018 · 2 次2026
机器学习与深度学习›优化与可微编程·arXiv:1810.08297 · cs.MS

Dynamic Automatic Differentiation of GPU Broadcast Kernels

Jarrett Revels, Tim Besard, Valentin Churavy, Bjorn De Sutter, Juan Pablo Vielma · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-10-23 · 最近 2024-06-18
2不同的人
4活跃月份
4讨论串
6被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We show how forward-mode automatic differentiation (AD) can be employed within larger reverse-mode computations to dynamically differentiate broadcast operations in a GPU-friendly manner. Our technique fully exploits the broadcast Jacobian's inherent sparsity structure, and unlike a pure reverse-mode approach, this "mixed-mode" approach does not require a backwards pass over the broadcasted operation's subgraph, obviating the need for several reverse-mode-specific programmability restrictions on user-authored broadcast operations. Most notably, this approach allows broadcast fusion in primal code despite the presence of data-dependent control flow. We discuss an experiment in which a Julia implementation of our technique outperformed pure reverse-mode TensorFlow and Julia implementations for differentiating through broadcast operations within an HM-LSTM cell update calculation.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论jrevels · 2018-11

“Author here; the arxiv version can be found at https://arxiv.org/abs/1810.08297. Not much different from OP's linked version, but it includes citations to other interesting Julia AD/TPU-related papers that utilize this technique. Happy to answer any questions, at least until I turn in for the night:)”

Dynamic Automatic Differentiation of GPU Broadcast Kernels [pdf]原评论 ↗
评论mattjjatgoogle · 2020-03

“No worries! I didn't mean it as a correction so much as just a discussion; I'm sure it's true that other autodiff systems have very sophisticated automatic remat (like https://openreview.net/forum?id=BkYYXJ9i-). I'm hoping as users push JAX on new applications, especially in simulation and scientific computing, we'll learn a lot and be able to improve! T…”

Using JAX, numpy, and optimization techniques to improve separable image filters原评论 ↗
评论jrevels · 2024-06

“Cool stuff! Thanks for the write-up. Getting broadcast ergonomics right in a fully statically typed context seems both tricky and pretty satisfying. > We thought it’d be a fun/simple feature and a good break from the heavy duty work on automatic differentiation we had just published Perhaps it wasn't as much of a break as you might think - funnily enoug…”

NumPy-style broadcasting in Futhark原评论 ↗
被提交为 story · 1 次