“You might enjoy this paper if you haven't already read it.”
Neural Turing Machines
给神经网络配可微外部记忆,学会简单算法
把神经网络与可通过注意力机制读写的外部记忆耦合,类比图灵机或冯·诺依曼架构,但整体端到端可微,因此可用梯度下降训练。初步结果表明,神经图灵机能从输入输出样例中推断出复制、排序和联想回忆等简单算法。
评论者持续把它当作带外部记忆的神经网络和让网络学习算法的代表,用来讨论神经网络能否做算法推理,也承认其实验多为玩具问题。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮你建立 LSTM 序列建模、编码器-解码器和注意力机制的基础,神经图灵机的读写头寻址正是软注意力的延伸。
- 1
神经图灵机的控制器常用 LSTM,这篇给出 LSTM 序列建模的基础。
读理解 LSTM 如何通过隐状态做序列预测,作为对比:这类网络的记忆只存在于隐状态中。
- 2
编码器-解码器把序列压进定长向量,这一瓶颈正是外部记忆要解决的。
读抓编码器-解码器的结构和定长向量的信息瓶颈。
- 3
seq2seq 展示了 RNN 学映射序列,也暴露了长序列表示能力的限制。
读读模型结构与对长序列效果的讨论,与神经图灵机的复制任务对照。
- 4
软对齐的注意力给出可微的内容寻址思路,神经图灵机的读写寻址与之相通。
读读对齐模型:打分、softmax 权重、加权求和得到上下文向量。
- 5Neural Turing Machines
用可微的读写头把外部记忆接到控制器上,端到端学简单算法。
读重点读内容寻址与位置寻址、读写操作,以及复制、排序、联想回忆等实验。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Take a look at these two papers:”
“check out the NTM paper: Neural Turing Machines”
“But the paper was widely advertised in the mainstream press as being successful, even though it was not tested on”
“This concept was hyped up (because DeepMind) for a while at the time and in reality it amounted to nothing.”
“For arXiv submissions, please submit the abstract link; in this case: Neural Turing Machines”
看全部 20 条 →
“This is the basis of an architecture called a 'neural turing machine' and it is, indeed, Turing complete and can be trained through standard neural network training algorithms to perform arbitrary computations.”
“A fairly recent addition to deep nets is making use of memory: Neural Turing Machines "Neural Turing Machines".”
“And we have separate evidence that they can (in principle) capture semantic meaning and algorithmic reasoning as well, for example: Neural Turing Machines”
“based on Graves "Neural Turing Machines" Neural Turing Machines, which attempts to infer algorithms from the result.”
“For example, there's software that can learn some simple algorithms: Neural Turing Machines”
“Turing machines [8]”
“One related work that came to mind was Neural Turing Machines, which augments the LSTM network with a (fixed size) memory [1].”
“There is also research lately into weirder types of neural nets such as 'Neural Turing Machines'.”
“Here's the original paper on Neural Turing machines: Neural Turing Machines”
“In fact there is already a lot of solid work just on this subject:”
“How would you sort integers? using neural turing machines: Neural Turing Machines”
“Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.”
“Are you aware of the Neural Turing Machine”
“Interesting related work: Graves[0] trained a "neural Turing machine" to encode instructions for a basic sorting algorithm.”