“If you have a strong CS background, they'll take you a long way:”
Formal Algorithms for Transformers
用精确的伪代码和数学记号描述 Transformer 的架构与训练
这份文档力求自成一体、数学上精确地概述 Transformer 的架构与算法(不涉及实验结果),内容包括 Transformer 是什么、如何训练、用于什么任务、关键组件,以及主要模型的预览。读者需要熟悉基本的机器学习术语和 MLP 等较简单的网络结构。
HN 上的人推荐它是因为用伪代码和严格定义讲清了注意力层和各类 Transformer 变体(编码器-解码器与仅解码器),适合想看真正数学而非泛泛解释的人,也有人抱怨它的记号不寻常。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: Formal Algorithms for Transformers”
“I've been reading this paper with pseudocode for various transformers and finding it helfpul: Formal Algorithms for Transformers”
“I've been reading this paper with pseudocode for various transformers and finding it helfpul: Formal Algorithms for Transformers”
“A more recent academic but high-level explanation of transformers, very good for detail on the different flow flavors (e.g. encoder-decoder vs decoder only), is Formal Algorithms for Transformers[2], from DeepMind.”
“Like a lot of math, the notation is daunting, but once you become familiar with it, it really is a nice tool for thought.”
看全部 14 条 →
“This is a great transparent and clear description of what is actually going on in attention layers.”
“is a proper account of the architectures and what tasks they naturally lend themselves to”
“the paper that was most helpful for me was [Formal Algorithms for Transformers]”
“The paper provides mathematically precise definitions of all the parts of a transformers”
“I found Formal Algorithms for Transformers [1] (a Google paper from 2022) to have a great pedagogical style.”
“If you want the actual maths instead of handwaving, I recommend: Formal Algorithms for Transformers”
“formal algorithms for transformers provides perhaps the most succinct introduction to attention I can imagine”
“Interesting but hard to read since it uses a quite unique notations for matrix indexing and multplication. Why???”
- Formal Algorithms for Transformers ↗hexhowells · 2022-07-20 · 106 分