“For the rest, you'll want to read the paper: Linear Transformers Are Secretly Fast Weight Programmers”
Linear Transformers Are Secretly Fast Weight Programmers
证明线性注意力与上世纪 90 年代的快速权重编程器形式等价
证明线性化自注意力与 90 年代初的快速权重控制器形式上等价:慢速网络通过梯度下降学习,用键值外积序列去编程快速网络的权重。据此指出线性 softmax 注意力变体的记忆容量有限,把纯加性外积换成类 delta 规则的更新以便修正键值映射,并提出新的注意力线性化核函数,在合成检索、机器翻译和语言建模上验证了收益。
HN 上的人引用它说明 Transformer 可看作在编程其他网络的权重,并作为 DeltaNet 等线性 RNN 路线的源头,也涉及对 Schmidhuber 先行性的讨论。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Also if you like maths, equations (4)-(19) from Linear Transformers Are Secretly Fast Weight Programmers are a really spectacular example of using math when it's the right way to explain something, instead of using it to hide things.”
“Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before:”
“Linear Transformers Are Secretly Fast Weight Programmers, Linear Transformers Are Secretly Fast Weight Programmers”
“Transformers are just networks that learn to program the weights of other networks [1].”
“[3] Linear Transformers Are Secretly Fast Weight Programmers”