按年被提到
20232023 · 2 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2310.15916 · cs.CL

In-Context Learning Creates Task Vectors

Roee Hendel, Mor Geva, Amir Globerson · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-10-26 · 最近 2025-12-14
2不同的人
2活跃月份
3讨论串
66被引(OpenAlex)
arXiv 摘要(候选层不写导读)

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set $S$ to find a best-fitting function $f(x)$ in some hypothesis class. Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query $x$ and a single "task vector" calculated from the training set. Thus, ICL can be seen as compressing $S$ into a single task vector $\boldsymbolθ(S)$ and then using this task vector to modulate the transformer to produce the output. We support the above claim via comprehensive experiments across a range of models and tasks.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论og_kalu · 2023-10

“This is speculation but i don't think this information (how it came to be) is actually irrelevant information for the LLM. Your state, which transformer LLMs notably have no way of transferring is "just" the computation that preceded the prediction. I think LLMs use previous tokens to compute the likely state vector that preceded the tokens and make pred…”

LLMs playing chess are sensitive to how the position came to be原评论 ↗
评论potatoman22 · 2025-12

“It's technically deterministic, but it feels nondeterministic in chatbots since tokens are randomly sampled (temp > 0) and input is varied. Using the right prompt makes the model perform better on average, so it's not completely dumb. I like task vectors and soft prompts because I think they show how prompt engineering is cool and useful. https://arxiv.…”

Ask HN: How can I get better at using AI for programming?原评论 ↗
被提交为 story · 1 次