“These are both good reads if you find yourself tending to agree with the author”
Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
把上下文学习解释为注意力对梯度下降的对偶形式
论文把语言模型看作元优化器,把上下文学习理解为隐式微调:理论上,Transformer 注意力存在梯度下降的对偶形式,GPT 先根据示例产生元梯度,再把它们作用到原模型上形成上下文学习模型。在真实任务上对比上下文学习与显式微调,发现两者在多个角度行为相似;受对偶形式启发,作者还设计了带动量的注意力,性能优于普通注意力。
评论者反复引它作为证据,说明上下文学习近似于推理时的隐式梯度下降,LLM 内部做的计算远不止简单的下一词预测。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The idea in this technical paper is that with attention mechanism even in forward computation there is an inner state that is updated following a meta-gradient”
“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“There’s a recent paper that posits language models are meta learners where the transformer layer is approximating stochastic gradient descent updates from the inputs”
看全部 20 条 →
“Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers”
“Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers”
“(gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“(gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“In context learning is already implicit finetuning. Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers.”
“As [1[] shows, instead they perform very sophisticated calculations on their compressed knowledge representation to try to generate acceptable text.”
“Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers shows a LLM is doing gradient descent on the context window at inference time.”
“(gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“As is speculated in Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers, it's possible there's a relationship between the depth of the model and the length of chained rules it can learn”
“GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“Language Models Secretly Perform Gradient Descent as Meta-Optimizers: Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers”
“metrics in Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers (linked above) where they show that in-context learning actually changes the attention map in a similar way to traditional fine tuning”
“Eh no.”
“It has been shown that LLMs can perform e.g. gradient descent internally”