“What I understood: they assume a hypothetical bipartite graph with text nodes (like hypothetical paragraphs / chunks of texts) on one side, and “skill” nodes on the other. The skills could be things like: • ability to understand the word “because” [causality] • ability to divide two numbers • ability to detect irony Each text node is connected to all …”
A Theory for Emergence of Complex Skills in Language Models
A major driver of AI products today is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly understood, and a mechanistic explanation via mathematical analysis of gradient-based training seems difficult. The current paper takes a different approach, analysing emergence using the famous (and empirical) Scaling Laws of LLMs and a simple statistical framework. Contributions include: (a) A statistical framework that relates cross-entropy loss of LLMs to competence on the basic skills that underlie language tasks. (b) Mathematical analysis showing that the Scaling Laws imply a strong form of inductive bias that allows the pre-trained model to learn very efficiently. We informally call this {\em slingshot generalization} since naively viewed it appears to give competence levels at skills that violate usual generalization theory. (c) A key example of slingshot generalization, that competence at executing tasks involving $k$-tuples of skills emerges essentially at the same scaling and same rate as competence on the elementary skills themselves.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“This article is 3 years old, right before GPT-3's launch. So it doesn't have all the recent info. We're not using meta-learning much, the problem of using few examples to train a model has been solved by few-shot prompting, zero shot, or fine-tuning with a small dataset. The problem with meta-learning is that it tries to back-propagate through the inner l…”
“There is a Sanjeev Arora paper "A Theory for Emergence of Complex Skills in Language Models" (https://arxiv.org/pdf/2307.15936) on this subject. The key idea is there is cross entropy (how "surprised" the model is with the "correct" next token, lower is better), some of which is inherent in the language and therefore unavoidable, and the rest is model err…”