“Some additional interesting papers about it:”
Transformers learn in-context by gradient descent
证明线性自注意力可实现梯度下降,解释上下文学习的机制
论文提出,用自回归目标训练 Transformer 与基于梯度的元学习密切相关:先给出一个权重构造,证明单个线性自注意力层引起的数据变换与回归损失上的梯度下降等价。在简单回归任务上训练仅含自注意力的 Transformer,其学到的模型与梯度下降所得高度相似,甚至权重与该构造吻合,所以训练后的 Transformer 成为 mesa-optimizer,即在前向传播中用梯度下降学习模型。作者还讨论了它与 induction head 的联系。
评论者用它说明 LLM 的上下文学习内部可能在做类似梯度下降的事(mesa-optimizer),把它作为这一机制的证据来引用,也有人只把它当作一种声称。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The paper that made me unsure was "Transformers learn in-context by gradient descent" [1].”
“See work from Anthropic[1] and Google[2] that explores this.”
“by performing gradient descent within the overall network [1].”
“Presumably doing something that looks a bit like gradient decent, see Transformers learn in-context by gradient descent”
“This paper claimed transformers learn a gradient-descent mesa-optimizer as part of in-context learning”
看全部 7 条 →
“A few years ago it was discovered that LLMs can implement gradient descent in their processes.”
- Transformers learn in-context by gradient descent ↗sva_ · 2022-12-17 · 1 分