推荐tsurba · 2025-03
“For a modern take on RNNs, read Resurrecting Recurrent Neural Networks for Long Sequences by DeepMind.”
推荐 DeepMind 的这篇,作为 RNN 的现代解读
精心设计的线性循环单元 LRU 追平深度状态空间模型
深度状态空间模型(SSM)在长序列任务上表现出色,但它与 RNN 的性能差距来源并不清楚。论文用标准的信号传播论证来设计深层 RNN,通过消融线性化与对角化递归、更好的参数化与初始化、前向归一化等改动,找出性能提升的来源。最终得到的线性循环单元(LRU)在 Long Range Arena 上匹配 SSM 的性能和训练效率。
HN 上的人在讨论 Transformer 之外的长序列方案时提它,作为现代 RNN 以及线性循环网络出人意料地强的依据。
“For a modern take on RNNs, read Resurrecting Recurrent Neural Networks for Long Sequences by DeepMind.”
“"Resurrecting Recurrent Neural Networks for Long Sequences" Resurrecting Recurrent Neural Networks for Long Sequences”
“[1] shows how to train an RNN in a GPT-like mode, using parallel scan, with great performance on Path-X, which has a sequence length of 16k.”
“And linear recurrent networks are surprisingly powerful (Resurrecting Recurrent Neural Networks for Long Sequences, [OTHER_PAPER]).”