推荐hansonw · 2024-04
“This is also a good paper on the subject:”
推荐这篇同主题的好论文
用 RASP 猜想解释 Transformer 何时能做长度泛化
论文研究 Transformer 能否学到解决任务的真实算法,聚焦算法任务上的长度泛化。作者借助为 Transformer 计算模型设计的 RASP 语言,提出 RASP 泛化猜想:若任务能用一个对所有输入长度都适用的短 RASP 程序解决,Transformer 往往就能做长度泛化。该猜想涵盖了多数已知的长度泛化实例,并被用来大幅改进奇偶校验、加法等传统难题的泛化表现。
HN 上的人在争论 Transformer 是否只是记忆时引用它,说明 Transformer 能从例子中学到算法,且某些局限并非来自分词。
“This is also a good paper on the subject:”
“We can prove that transformers can do computation beyond memorization, and that they can learn it from examples, unless you think learning an algorithm is memorization.”
“Tokenization is often scapegoated for many transformer limitations.”