按年被提到
20212023 · 6 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2106.06981 · cs.LG

Thinking Like Transformers

像 Transformer 一样思考
Gail Weiss, Yoav Goldberg, Eran Yahav · 2021

提出编程语言 RASP,作为 Transformer 编码器的计算模型

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 11 人首次 2021-06-16 · 最近 2026-11-03
11不同的人
11活跃月份
13讨论串
21被引(OpenAlex)

循环网络对应有限状态机,而 Transformer 缺少这样熟悉的计算类比。作者把 Transformer 编码器的注意力和前馈计算映射成简单原语,构成受限访问序列处理语言(RASP),并展示如何用它编写直方图、排序和 Dyck 语言的解法,以及如何训练 Transformer 去模仿 RASP 程序。分析 RASP 程序还能推出完成某任务至少需要的注意力头数和层数,并可用这个抽象解释近期工作中观察到的现象。

HN 为什么提它

HN 上的人把它推荐为读完 Attention 之后的第二篇论文,用它说明 Transformer 能计数、排序,以及逻辑计算可被编译为 Transformer 权重;也有人质疑它的主张。

奠基之作难度 4 / 5约 50 分钟经得起时间
读之前最好知道
1Transformer 编码器与注意力机制
2形式语言与 Dyck 语言
3有限状态机与计算模型
4序列到序列的编程原语(select / aggregate)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐lalaithion · 2021-12

“a good second paper to read is Thinking Like Transformers”

推荐作为读完 Attention 之后的第二篇论文
The potential of transformers in reinforcement learning原评论 ↗
推荐nikvaes · 2024-04

“This paper indeed seems a promising direction.”

认为这篇论文的方向很有前景
Mixture-of-Depths: Dynamically allocating compute in transformers原评论 ↗
批评YeGoblynQueenne · 2023-06

“That's one preprint on arxiv, that makes a wild claim about a new concept that they acronymise as "RASP".”

From Word Models to World Models原评论 ↗
引用attemptone · 2022-10

“The paper suggests that transformer work on a set of select, aggregate and element-wise operations.”

引用它说明 Transformer 基于 select、aggregate 和逐元素运算
GitHub Copi­lot inves­ti­ga­tion原评论 ↗
引用jbay808 · 2023-05

“Counting is a task that transformers can do, per Weiss.[1]”

引用 Weiss 的论文说明 Transformer 能做计数
It's not just statistics: GPT-4 does reason原评论 ↗
引用jbay808 · 2023-06

“Transformers are RASP programs, which includes sorting programs. See the Weiss paper.”

指出 Transformer 就是 RASP 程序,包括排序程序
From Word Models to World Models原评论 ↗
看全部 12 条 →
引用jbay808 · 2023-08

“transformers are capable of directly representing sorting programs.”

引用它说明 Transformer 能直接表示排序程序
Graph of Thoughts: Solving Elaborate Problems with Large Language Models原评论 ↗
引用teddykoker · 2023-09

“They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components”

介绍它提出的原始编程语言 RASP,可用 Transformer 组件建模
I made a transformer by hand (no training)原评论 ↗
引用astrange · 2025-11

“An LLM can perform nearly arbitrary computation inside its fixed-size memory.”

借此论证 LLM 能在固定大小内存里执行近乎任意的计算
Ask HN: How are Markov chains so different from tiny LLMs?原评论 ↗
引用leogao · 2025-11

“There's been some work (e.g RASP - Thinking Like Transformers) on taking logical computations and compiling them into transformer weights.”

引用 RASP 说明可把逻辑计算编译成 Transformer 权重
Weight-sparse transformers have interpretable circuits [pdf]原评论 ↗
引用andy12_ · 2026-02

“A normal traditional program, like a calculator, encoded as an autoregressive transformer that calculates its output one word at a time (compiled neural networks) [1][2]”

引用它说明传统程序可编译成自回归 Transformer
I want to wash my car. The car wash is 50 meters away. Should I walk or drive?原评论 ↗
引用syllogism · 2026-11

“Here's the work that shows how rule-based logic can be compiled into transformer weights:”

拿它说明基于规则的逻辑可以编译进 Transformer 权重
The 'Astra can read and write weights to improve Qwen' post is misleading原评论 ↗
被提交为 story · 2 次