“most of the literature I've seen is pretty application-specific, but try [1][2] for flavour.”
Sequence to Sequence Learning with Neural Networks
用两个 LSTM 编码再解码,实现端到端序列映射
论文提出一种对序列结构几乎不做假设的端到端序列学习方法:用多层 LSTM 把输入序列映射成固定维度的向量,再用另一个深度 LSTM 从该向量解码出目标序列。在 WMT'14 英法翻译上,LSTM 的 BLEU 为 34.8,高于短语式 SMT 系统的 33.3;用 LSTM 对 SMT 的 1000 个候选重排后升到 36.5。把源句的词序反转(目标句不反转)会明显改善效果,因为这引入了许多源与目标之间的短期依赖,使优化更容易。
评论者把它当作神经机器翻译和「思想向量」的奠基论文反复引用,用来解释用固定向量编码整句、输入反转技巧,也提到它早于 OpenAI 成立。
学习路径
读这篇之前,按顺序读书架上的 2 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮你先理解用 LSTM 逐步生成序列,以及循环网络做编码器-解码器的早期尝试,再读 seq2seq。
- 1
seq2seq 的解码器就是 LSTM 逐步预测下一个符号的生成过程,来自这里的做法。
读LSTM 如何逐点预测并生成序列;next-step prediction 的训练方式。
- 2
几乎同期提出的 RNN 编码器-解码器,把输入压成定长向量再解码,与 seq2seq 思路平行。
读编码器-解码器结构及其用于短语表示的做法;与 seq2seq 对比二者的差别。
- 3Sequence to Sequence Learning with Neural Networks
用深层 LSTM 编码再解码,把端到端序列到序列学习推到机器翻译上。
读整体模型结构、反转源句的技巧,以及翻译任务的结果。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“New neural machine translation architectures are experimenting with pairs of neural encoders / decoders,”
“which allows whole sentences to be digested and translated.”
“Seq2seq as an approach does achieve close to state-of-the-art for translation[1].”
“[2] The paper Geoff Hinton is reffering to: Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, Quoc V. Le Sequence to Sequence Learning with Neural Networks”
“I think it was with the seq2seq paper of Sutskever, Vinyals and Le in 2014:”
看全部 9 条 →
“The encoder encoded the input as a single vector and the decoder would decode this into a sentence (eg. for machine translation) [1].”
“Failing that, Seq2seq predates OpenAI's incorporation:”
“So, you might try doing something like reversing the input to force learning to take a less crappy path [0].”