“An example of this is neural translation with the Attention is all you need paper, which is the top of the line in translation.”
Attention Is All You Need
提出完全基于注意力机制的 Transformer 架构,取代循环与卷积
论文提出 Transformer,一种只依赖注意力机制、完全不用循环和卷积的序列转换网络。在 WMT 2014 英德和英法机器翻译任务上,它的质量超过当时的最好结果,并且更易并行、训练成本明显更低;它在英语成分句法分析上也表现良好。
它被反复称为现代大语言模型的奠基论文,HN 上的人把它当作理解 Transformer 架构的原始出处和入门起点,也有人说原文其实并不好读。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先理解序列到序列框架和注意力机制如何解决定长向量瓶颈,再了解残差连接和当时的 LSTM 翻译系统,就能看清 Transformer 要取代的是什么。
- 1
Transformer 沿用的编码器-解码器序列转换框架来自这里
读编码器-解码器结构与端到端训练;理解它的定长向量瓶颈
- 2
Transformer 的注意力思想源于这里的软对齐
读第 3 节的对齐模型与上下文向量计算;理解注意力权重的含义
- 3
Transformer 要超越的 LSTM+注意力的工业级基线
读8 层 LSTM 编码器-解码器结构与残差连接;对照 Transformer 的 BLEU 和训练成本
- 4
Transformer 每个子层都使用残差连接
读残差块 y=F(x)+x 的思想与它为何便于训练深层网络;跳过 ImageNet 实验
- 5Attention Is All You Need
目标论文:用纯注意力取代循环和卷积
读第 3 节的整体架构、缩放点积注意力、多头注意力和位置编码;第 4 节说明为何选自注意力
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“like having read the seminal Transformer paper,”
“Read the original paper on the architecture: Attention Is All You Need”
“I think this is the best paper to read to get an understanding of this transformer architecture.”
“The original paper that introduced the Transformer architecture is quite accessible and outlines a lot of the history and rational for the design [1].”
“Read the intro in the original paper "Attention is all you need paper"”
看全部 20 条 →
“Have you read the 2017 paper Attention is All You Need? It's not a book, but it is long form, a few math formulas but not too many.”
“The most notable paper is probably the one that defined the concept of transformers, Attention Is All You Need: Attention Is All You Need”
“If you want to learn more, here's Vaswani et al 2017 - the seminal paper Attention Is All You Need”
“[0]: The original paper: Attention Is All You Need”
“Here's some other works you might never have read, based on their titles.”
“here are the most helpful resources for understanding transformer basics in particular:”
“The abstract Attention Is All You Need explains it well:”
“This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017.”
“you should read, is sparked an entire new AI dawn, it might answer your question”
“I think they need to be looked into more. They just work too well.”
“I found the original paper to be excellent, but most everyone during my PhD, including my advisor, found it to be inscrutable”
“Google Brain outperforms LSTMs with Convolutional Networks in speed and accuracy, seeming to confirm LSTMs are not optimal for NLP at least:”
“Googles Transformer was made public in June 2017, in the paper Attention is all you need, Attention Is All You Need”
“One of the core operations of the transformer network[1] is a (LxL) x (LxE) matrix multiply”