“a good second paper to read is Thinking Like Transformers”
Thinking Like Transformers
提出编程语言 RASP,作为 Transformer 编码器的计算模型
循环网络对应有限状态机,而 Transformer 缺少这样熟悉的计算类比。作者把 Transformer 编码器的注意力和前馈计算映射成简单原语,构成受限访问序列处理语言(RASP),并展示如何用它编写直方图、排序和 Dyck 语言的解法,以及如何训练 Transformer 去模仿 RASP 程序。分析 RASP 程序还能推出完成某任务至少需要的注意力头数和层数,并可用这个抽象解释近期工作中观察到的现象。
HN 上的人把它推荐为读完 Attention 之后的第二篇论文,用它说明 Transformer 能计数、排序,以及逻辑计算可被编译为 Transformer 权重;也有人质疑它的主张。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“This paper indeed seems a promising direction.”
“That's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP".”
“The paper suggests that transformer work on a set of select, aggregate and element-wise operations.”
“Counting is a task that transformers can do, per Weiss.[1]”
“Transformers are RASP programs, which includes sorting programs. See the Weiss paper.”
看全部 12 条 →
“transformers are capable of directly representing sorting programs.”
“They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components”
“An LLM can perform nearly arbitrary computation inside its fixed-size memory.”
“There's been some work (e.g RASP - Thinking Like Transformers) on taking logical computations and compiling them into transformer weights.”
“A normal traditional program, like a calculator, encoded as an autoregressive transformer that calculates its output one word at a time (compiled neural networks) [1][2]”
“Here's the work that shows how rule-based logic can be compiled into transformer weights:”
- Thinking Like Transformers ↗ArtWomb · 2021-06-16 · 161 分
- Thinking Like Transformers (pdf) ↗jbay808 · 2023-06-13 · 2 分