按年被提到
20222023 · 5 次2026
大模型与语言模型›可解释性与表征研究·arXiv:2211.15661 · cs.LG

What learning algorithm is in-context learning? Investigations with linear models

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, Denny Zhou · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 5 人首次 2023-02-10 · 最近 2023-04-08
5不同的人
2活跃月份
4讨论串
138被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Neural sequence models, especially transformers, exhibit a remarkable capacity for in-context learning. They can construct new predictors from sequences of labeled examples $(x, f(x))$ presented in the input without further parameter updates. We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context. Using linear regression as a prototypical problem, we offer three sources of evidence for this hypothesis. First, we prove by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. Second, we show that trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression, transitioning between different predictors as transformer depth and dataset noise vary, and converging to Bayesian estimators for large widths and depths. Third, we present preliminary evidence that in-context learners share algorithmic features with these predictors: learners' late layers non-linearly encode weight vectors and moment matrices. These results suggest that in-context learning is understandable in algorithmic terms, and that (at least in the linear case) learners may rediscover standard estimation algorithms. Code and reference implementations are released at https://github.com/ekinakyurek/google-research/blob/master/incontext.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论dr_dshiv · 2023-02

“The author says: “Large Language Models don’t understand anything — and they can’t get better at something they can’t do.” Yet, LLM’s can understand (unless one claims that understanding requires consciousness): https://arxiv.org/abs/2101.06573 And, strangely enough, LLM’s can get better at something they can’t do. https://news.mit.edu/2023/large-langu…”

ChatGPT Hype Is Proof Nobody Understands AI原评论 ↗
评论benlivengood · 2023-02

“The article doesn't understand large language models. It makes an assumption about how they work and uses that assumption to draw a conclusion. Most likely, as found in small models [0], large autoregressive transformers understand machine learning algorithms well enough to apply them to their inputs to achieve in-context learning (few-shot prompting) by…”

ChatGPT Hype Is Proof Nobody Understands AI原评论 ↗
评论tehf0x · 2023-04

“Ah the good old "it's not me it's the test" argument. These systems are not just next token predictors, they learn complex algorithms and can perform general computation, its just so happens that by asking them to next-token predict the internet they learn a bunch of smart ways to compress everything, potentially in a way similar to how we might use a gen…”

GPT-4 Takes a New Midterm and Gets an A原评论 ↗
被提交为 story · 2 次