“consider this paper”
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
线性注意力把 Transformer 改写成 RNN,长序列推理极快
论文把自注意力写成核特征映射的线性点积,并利用矩阵乘法的结合律把复杂度从 O(N²) 降到 O(N)。这种形式允许迭代实现,从而让自回归 Transformer 大幅加速,并揭示出它与循环神经网络的等价关系。线性 Transformer 性能与标准 Transformer 相近,在超长序列的自回归预测上最多快 4000 倍。
HN 上的人把它当作“Transformer 可视为带内部状态的 RNN”的出处(尤其是第 3.4 节),在谈 RWKV、Mamba 等线性循环模型时反复追溯到它。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“(Transformers are RNNs Transformers are RNNs)”
“"Transformers are RNNs" Transformers are RNNs (see section 3.4)”
“Transformers are RNNs: section 3.4 (page 5) Transformers are RNNs”
“See above comment and "Transformers are RNNs" paper to convince yourself.”
“Additionally, Transformers are RNNs: Transformers are RNNs”
看全部 11 条 →
“the state-space transformation can be expressed as a form of causal linear attention[a] by compounding coefficients a_1... a_t at each time step t.”
“the state-space transformation can be expressed as a form of causal linear attention.[a]”
“[2] Transformers are RNNs”
“* RWKV: Transformers are RNNs / [OTHER_PAPER] [OTHER_PAPER]”
“This is a form of linear attention that approximates standard scaled dot-product attention to arbitrary precision, by adding Taylor terms in an efficient manner.”
- Faster Autoregressive Transformers with Linear Attention ↗fofoz · 2020-07-03 · 6 分