“most of the literature I've seen is pretty application-specific, but try [1][2] for flavour.”
Neural Machine Translation by Jointly Learning to Align and Translate
为神经翻译引入软对齐的注意力机制,突破定长向量瓶颈
作者推测编码器-解码器把源句压成定长向量是性能瓶颈,提出让模型自动(软)搜索与当前目标词相关的源句部分,而无需显式切分。在英法翻译上,其性能与当时最好的基于短语的系统相当,且学到的软对齐符合人的直觉。
HN 上的人把它当作注意力机制的源头,推荐想理解 Transformer 或 cross-attention 的人先读它,因为它的原始注意力更简单易懂。
学习路径
读这篇之前,按顺序读书架上的 3 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先理解 LSTM 序列生成和编码器-解码器的结构,以及固定长度向量瓶颈,再读注意力如何解决这个问题。
- 1
LSTM 做序列生成的基础,目标论文的编码器与解码器都基于循环网络。
读LSTM 逐步预测与序列建模的思路;不必看手写体合成部分。
- 2
提出 RNN 编码器-解码器并带来 GRU,目标论文直接在这个框架上改进。
读编码器-解码器结构与 GRU 单元。
- 3
同期的序列到序列模型,把整句压成定长向量,正是目标论文指出的瓶颈。
读两个 LSTM 编码再解码的结构,以及长句性能下降的现象。
- 4Neural Machine Translation by Jointly Learning to Align and Translate
目标论文:用软对齐让解码器每一步关注源句不同位置。
读对齐模型与上下文向量的计算;双向 RNN 编码器;注意力可视化对齐图。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“learn the concept of attention, specifically cross attention first. And that is this paper:”
“text to text translation [6]”
“one of the earliest and coolest examples are neural attention mechanisms learning to align words between languages without any help[1]!”
“Attention was first coined in this paper (as far as I know):”
“a well known LSTM extension that worked well for machine translation (2014, Neural Machine Translation by Jointly Learning to Align and Translate)”
看全部 11 条 →
“It was really helpful to get a sense of the original 2014 concept of attention which is easier to understand but less powerful”
“but Neural Machine Translation by Jointly Learning to Align and Translate from 2014 is a famous one.”
“People did something similar to what you are describing 10 years ago: Neural Machine Translation by Jointly Learning to Align and Translate”
“Re: Machine Translation, note that Transformers were introduced for this task, and built on one of the earlier notions of attention in sequence models: Neural Machine Translation by Jointly Learning to Align and Translate (2014, 38k citations)”
“Neural Machine Translation by Jointly Learning to Align and Translate - Neural Machine Translation by Jointly Learning to Align and Translate”
- Neural Machine Translation by Jointly Learning to Align and Translate ↗__Joker · 2015-01-25 · 2 分
- Neural Machine Translation by Jointly Learning to Align and Translate (2015) ↗tosh · 2023-04-03 · 1 分