按年被提到
20222023 · 10 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2207.09238 · cs.LG

Formal Algorithms for Transformers

Transformer 的形式化算法
Mary Phuong, Marcus Hutter · 2022

用精确的伪代码和数学记号描述 Transformer 的架构与训练

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 12 人首次 2022-07-20 · 最近 2026-11-03
12不同的人
10活跃月份
14讨论串
83被引(OpenAlex)

这份文档力求自成一体、数学上精确地概述 Transformer 的架构与算法(不涉及实验结果),内容包括 Transformer 是什么、如何训练、用于什么任务、关键组件,以及主要模型的预览。读者需要熟悉基本的机器学习术语和 MLP 等较简单的网络结构。

HN 为什么提它

HN 上的人推荐它是因为用伪代码和严格定义讲清了注意力层和各类 Transformer 变体(编码器-解码器与仅解码器),适合想看真正数学而非泛泛解释的人,也有人抱怨它的记号不寻常。

教程难度 3 / 5约 90 分钟经得起时间
读之前最好知道
1基本机器学习术语
2多层感知机(MLP)等简单神经网络
3线性代数与矩阵记号
4伪代码(pseudocode)阅读

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐heyitsguay · 2023-01

“If you have a strong CS background, they'll take you a long way:”

推荐它,有较强 CS 背景的人读来受益很大
The Transformer Family原评论 ↗
推荐bfg_damien · 2023-04

“Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: Formal Algorithms for Transformers”

推荐这篇 DeepMind 论文,给出 Transformer 重要算法的伪代码
Ask HN: How to Learn LLM Fundamentals原评论 ↗
推荐Buttons840 · 2023-04

“I've been reading this paper with pseudocode for various transformers and finding it helfpul: Formal Algorithms for Transformers”

推荐它,自己读后觉得其中各种 Transformer 的伪代码很有帮助
A Baby GPT原评论 ↗
推荐Buttons840 · 2023-04

“I've been reading this paper with pseudocode for various transformers and finding it helfpul: Formal Algorithms for Transformers”

推荐它,读后觉得其中各种 Transformer 的伪代码很有帮助
What Are Transformer Models and How Do They Work?原评论 ↗
推荐dsubburam · 2023-04

“A more recent academic but high-level explanation of transformers, very good for detail on the different flow flavors (e.g. encoder-decoder vs decoder only), is Formal Algorithms for Transformers[2], from DeepMind.”

推荐它作为较新的高层学术讲解,讲清 encoder-decoder 与 decoder-only 的区别
Transformers from Scratch原评论 ↗
推荐Buttons840 · 2023-05

“Like a lot of math, the notation is daunting, but once you become familiar with it, it really is a nice tool for thought.”

推荐它,说其记号虽吓人,熟悉后是很好的思考工具
Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper原评论 ↗
看全部 14 条 →
推荐tomasff · 2023-06

“This is a great transparent and clear description of what is actually going on in attention layers.”

推荐它,认为清晰透明地讲出了 attention 层到底在做什么
Attention is Turing complete [pdf]原评论 ↗
推荐dsubburam · 2023-07

“is a proper account of the architectures and what tasks they naturally lend themselves to”

推荐它,认为准确讲清了各种架构及其适合的任务
What We Know About LLMs (A Primer)原评论 ↗
推荐svcrunch · 2023-12

“the paper that was most helpful for me was [Formal Algorithms for Transformers]”

推荐它,称对自己帮助最大,伪代码清晰精确
"Attention", "Transformers", in Neural Network "Large Language Models"原评论 ↗
推荐rck · 2024-01

“The paper provides mathematically precise definitions of all the parts of a transformers”

推荐它,给出 Transformer 各部分在数学上精确的定义
Relearning math as an adult原评论 ↗
推荐sharma-arjun · 2026-06

“I found Formal Algorithms for Transformers [1] (a Google paper from 2022) to have a great pedagogical style.”

推荐它,觉得教学风格很好
How LLMs Work原评论 ↗
推荐wren6991 · 2026-06

“If you want the actual maths instead of handwaving, I recommend: Formal Algorithms for Transformers”

推荐它,想看真正的数学而非空谈时可读
Can I Buy Your KV Cache?原评论 ↗
推荐horseloverthin · 2026-11

“formal algorithms for transformers provides perhaps the most succinct introduction to attention I can imagine”

推荐它作为 attention 最简洁的入门介绍
The Little Book of Generative AI Foundations: An Intuitive Mathematical Primer原评论 ↗
批评_giorgio_ · 2023-12

“Interesting but hard to read since it uses a quite unique notations for matrix indexing and multplication. Why???”

"Attention", "Transformers", in Neural Network "Large Language Models"原评论 ↗
被提交为 story · 1 次