“Worth considering: Relating transformers to models and neural representations of the hippocampal formation”
Relating transformers to models and neural representations of the hippocampal formation
带循环位置编码的 Transformer 能复现海马的位置细胞与网格细胞
作者表明,配备循环位置编码的 Transformer 能复现海马结构中精细调谐的空间表征,尤其是位置细胞和网格细胞,并说明这并不意外,因为它与神经科学中现有的海马模型密切相关。文中还显示 Transformer 版本相比神经科学版本有大幅的性能提升,并据此推测更广的皮层区域如何完成语言理解等复杂任务。
评论者在讨论 Transformer 与大脑的相似性、位置编码或可解释模型时引用它,作为人工网络和海马模型共享结构的证据,也有人对其说法持怀疑态度。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“You might enjoy this paper[0] which shows that recurrent position encodings recover grid cell representations and maps to path integration found in a popular model of the hippocampus.”
“I also recommend Relating transformers to models and neural representations of the hippocampal formation, but, in short, transformer models have functions and emergent structures”
“might make it more relevant though”
“Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that.”
“solving a similar problem to Relating transformers to models and neural representations of the hippocampal formation, but with the additional constraint of forward functional interpretability”
- Relating Transformers to Modes and Neural Representations of the Hippocampal ↗crescit_eundo · 2022-04-11 · 3 分
- Relating Transformers to Neural Representations of the Hippocampal Formation ↗beefman · 2022-04-12 · 1 分
- Relating transformers to neural representations of hippocampal formation ↗tech-sucker · 2022-05-23 · 2 分