按年被提到
20222023 · 19 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2212.10559 · cs.CL

Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers

GPT 为什么能上下文学习?语言模型隐式地把梯度下降当作元优化器
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, Furu Wei · 2022

把上下文学习解释为注意力对梯度下降的对偶形式

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 11 人首次 2022-12-21 · 最近 2024-07-07
11不同的人
7活跃月份
17讨论串
90被引(OpenAlex)

论文把语言模型看作元优化器,把上下文学习理解为隐式微调:理论上,Transformer 注意力存在梯度下降的对偶形式,GPT 先根据示例产生元梯度,再把它们作用到原模型上形成上下文学习模型。在真实任务上对比上下文学习与显式微调,发现两者在多个角度行为相似;受对偶形式启发,作者还设计了带动量的注意力,性能优于普通注意力。

HN 为什么提它

评论者反复引它作为证据,说明上下文学习近似于推理时的隐式梯度下降,LLM 内部做的计算远不止简单的下一词预测。

其他难度 4 / 5约 40 分钟经得起时间
读之前最好知道
1Transformer 与自注意力
2上下文学习(in-context learning)与少样本提示
3梯度下降与微调
4线性注意力

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐dhruvdh · 2023-02

“These are both good reads if you find yourself tending to agree with the author”

推荐它,把上下文学习解释为隐式梯度下降,同意作者观点的人值得一读
ChatGPT Is a Blurry JPEG of the Web原评论 ↗
引用GistNoesis · 2023-01

“The idea in this technical paper is that with attention mechanism even in forward computation there is an inner state that is updated following a meta-gradient”

解释其思路:注意力机制在前向计算中也有按元梯度更新的内部状态
The Expanding Dark Forest and Generative AI原评论 ↗
引用spion · 2023-02

“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

拿它说明 GPT 类模型内部实现了一个迷你的神经网络训练算法,即梯度下降微调
What Is ChatGPT Doing and Why Does It Work?原评论 ↗
引用kilgnad · 2023-02

“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

拿它说明 GPT 类模型内部实现了迷你训练算法,即对给定样例做梯度下降微调
To Build Truly Intelligent Machines, Teach Them Cause and Effect原评论 ↗
引用kilgnad · 2023-03

“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

拿它说明 GPT 类模型内部实现了迷你训练算法,即梯度下降微调
To Build Truly Intelligent Machines, Teach Them Cause and Effect原评论 ↗
引用cgearhart · 2023-03

“There’s a recent paper that posits language models are meta learners where the transformer layer is approximating stochastic gradient descent updates from the inputs”

提到有论文认为语言模型是元学习者,Transformer 层在近似 SGD 更新
Prompt to bypass the restrictions of Bing Chat or to restore the old “Sydney”原评论 ↗
看全部 20 条 →
引用nl · 2023-03

“Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers”

直接贴出标题,作为语言模型隐式做梯度下降的证据
Large language models are having their Stable Diffusion moment原评论 ↗
引用nl · 2023-03

“Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers”

贴出论文标题,主张语言模型暗中把梯度下降当作元优化器
引用HervalFreire · 2023-03

“(gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

引它说明 GPT 模型在上下文里内部实现了梯度下降
GPT-4 Designed a Programming Language原评论 ↗
引用HervalFreire · 2023-03

“(gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

引它说明 GPT 模型在上下文里内部实现了梯度下降
GPT-4 Designed a Programming Language原评论 ↗
引用og_kalu · 2023-03

“In context learning is already implicit finetuning. Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers.”

引它主张上下文学习本来就是隐式微调
OK, it’s time to freak out about AI原评论 ↗
引用nl · 2023-03

“As [1[] shows, instead they perform very sophisticated calculations on their compressed knowledge representation to try to generate acceptable text.”

引它说明 LLM 会在压缩的知识表示上做复杂计算,而非只是预测下一个 token
OK, it’s time to freak out about AI原评论 ↗
引用nl · 2023-03

“Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers shows a LLM is doing gradient descent on the context window at inference time.”

拿它说明 LLM 在推理时对上下文窗口做梯度下降
Sparks of Artificial General Intelligence: Early Experiments with GPT-4原评论 ↗
引用HervalFreire · 2023-03

“(gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

引它说明 GPT 模型在上下文里内部实现了梯度下降
Do large language models need sensory grounding for meaning and understanding?原评论 ↗
引用nl · 2023-03

“As is speculated in Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers, it's possible there's a relationship between the depth of the model and the length of chained rules it can learn”

引它推测模型深度可能与所能学到的链式规则长度有关
Cerebras-GPT vs. LLaMA AI Model Performance Comparison原评论 ↗
引用Herval_freire · 2023-03

“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

拿它说明 GPT 类模型内部实现了迷你训练算法,即梯度下降微调
AI-enhanced development makes me more ambitious with my projects原评论 ↗
引用nl · 2023-04

“Language Models Secretly Perform Gradient Descent as Meta-Optimizers: Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”

贴出它,作为语言模型暗中做梯度下降元优化的出处
Scaling Transformer to 1M tokens and beyond with RMT原评论 ↗
引用nl · 2023-04

“metrics in Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers (linked above) where they show that in-context learning actually changes the attention map in a similar way to traditional fine tuning”

引它的指标说明上下文学习改变注意力图的方式与传统微调相似
Scaling Transformer to 1M tokens and beyond with RMT原评论 ↗
引用og_kalu · 2023-05

“Eh no.”

简短否定,以此反对上下文学习等同于隐式梯度下降的说法
Chatbots Don’t Know What Stuff Isn’t原评论 ↗
引用sigmoid10 · 2024-07

“It has been shown that LLMs can perform e.g. gradient descent internally”

指出已有研究表明 LLM 内部可以做梯度下降之类的事
Reasoning in Large Language Models: A Geometric Perspective原评论 ↗