批评NitpickLawyer · 2025-08
“A depth of 4 is very small. It is very much a toy model.”
证明深度随输入长度对数增长的 Transformer 表达力更强
论文分析深度随上下文长度 n 以 Θ(log n) 增长的 Transformer,证明即便是高度一致(uniform)的这类模型,也能表达正则语言识别与图连通性这两个问题;在标准复杂度猜想下,固定深度 Transformer 无法表达它们。理论还定量预测了深度应如何随输入长度增长,并表明扩展深度比扩展宽度或增加思维链步数更高效。实验显示识别正则语言所需的理论深度与实际训练所需深度相当吻合。
评论者把它作为 Transformer 深度与推理能力关系的理论依据来引用,也有人批评实验里深度 4 的模型太小、像玩具模型。
“A depth of 4 is very small. It is very much a toy model.”
“sufficiently deep transformers with chain of thought have been shown to be theoretically capable of solving a very large class of problems.”
“because of A Little Depth Goes a Long Way.”
“There is even theoretical justification for why: A Little Depth Goes a Long Way”
“Will Merrill (again) has a paper here that discusses exactly this.”