引用336f5 · 2015-08
“Very deep NNs, which try to bypass the training difficulties in propagating error all the way from the bottom to the top”
用它说明极深网络靠跨层捷径绕开误差传播的训练困难
用门控单元让上百层的网络能直接用 SGD 训练
网络越深越难训练,极深网络的训练当时仍是开放问题。这篇扩展摘要提出 highway 网络架构,用可学习的门控单元调节信息在网络中的流动,让信息能在多层之间不受阻碍地沿“信息高速公路”传递;数百层的 highway 网络可直接用随机梯度下降训练,并适用于多种激活函数。
评论者把 highway 层与 LSTM、GRU、批归一化、残差块并列为深度网络的标准构件,并提到 Schmidhuber 实验室提出了更一般的“旁路”思路来训练深网。
“Very deep NNs, which try to bypass the training difficulties in propagating error all the way from the bottom to the top”
“higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5]”
“Ironically Schmidhuber's lab also came up with a more general version of bypassing Highway Networks to train deep networks”