按年被提到
20192020 · 1 次2026
机器学习与深度学习›神经网络架构探索·arXiv:1905.11786 · cs.LG

Putting An End to End-to-End: Gradient-Isolated Learning of Representations

Sindy Löwe, Peter O'Connor, Bastiaan S. Veeling · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-03-03 · 最近 2023-01-12
2不同的人
2活跃月份
2讨论串
84被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We propose a novel deep learning method for local self-supervised representation learning that does not require labels nor end-to-end backpropagation but exploits the natural order in data instead. Inspired by the observation that biological neural networks appear to learn without backpropagating a global error signal, we split a deep neural network into a stack of gradient-isolated modules. Each module is trained to maximally preserve the information of its inputs using the InfoNCE bound from Oord et al. [2018]. Despite this greedy training, we demonstrate that each module improves upon the output of its predecessor, and that the representations created by the top module yield highly competitive results on downstream classification tasks in the audio and visual domain. The proposal enables optimizing modules asynchronously, allowing large-scale distributed training of very deep neural networks on unlabelled datasets.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论LudwigNagasena · 2023-01

“The question was "If it's not new or novel, why aren't people using it?". For example, take a look at this paper: https://arxiv.org/abs/1905.11786. It was published 3 years ago, it also does parallel layer-wise optimization, it even talks about being inspired by biology, though the objective function is different from Hinton's. Why aren't people using it?…”

Deep Learning Pioneer Geoffrey Hinton Publishes New Deep Learning Algorithm原评论 ↗
被提交为 story · 1 次