“LLMs are a different beast in the ML world. Finetuned Palm for Medicine and Finetuned Minerva for Math all perform a good deal worse than GPT-4. A fine-tuned smaller model is by no means guaranteed to beat a larger more general one (though of course you may get acceptable performance). And then the necessity of fine-tuning itself is called into questio…”
Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models
We investigate how to elicit compositional generalization capabilities in large language models (LLMs). Compositional generalization empowers LLMs to solve complex problems by combining foundational skills, a critical reasoning ability akin to human intelligence. However, even the most advanced LLMs currently struggle with this form of reasoning. We examine this problem within the framework of in-context learning and find that demonstrating both foundational skills and compositional examples grounded in these skills within the same prompt context is crucial. We refer to this prompt structure as skills-in-context (SKiC). With as few as two exemplars, this in-context learning structure enables LLMs to tackle more challenging problems requiring innovative skill combinations, achieving near-perfect systematic generalization across a broad range of tasks. Intriguingly, SKiC also unlocks the latent potential of LLMs, allowing them to more actively utilize pre-existing internal skills acquired during earlier pretraining stages to solve complex reasoning problems. The SKiC structure is robust across different skill constructions and exemplar choices and demonstrates strong transferability to new tasks. Finally, inspired by our in-context learning study, we show that fine-tuning LLMs with SKiC-style data can elicit zero-shot weak-to-strong generalization, enabling the models to solve much harder problems directly with standard prompting.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“> Numbers (LLMs still can't do math and the ones that "can" are very brute forced and don't generalize) These seem to generalize. https://arxiv.org/abs/2211.09066 https://arxiv.org/abs/2308.00304 > Commutative properties (The (idk why anyone was surprised) "reversal curse": https://arxiv.org/abs/2309.12288) This was a retrieval from training issue no…”
“>Every human is not even capable of reading, or speaking, or thinking at all. But those humans who can't are not the benchmark of general human intelligence. It's simple. If you're going to disqualify a potential general intelligence because of a particular task solving ability, it better be something every general intelligence can do. Now let's forget …”
“the extraordinary claim would be that LLMs can only do things they've seen before exactly, given the compositional and emergent capabilities we observe. The evidence suggests they can generalize beyond their training in meaningful ways, even if imperfectly...if a human came out living but with a brain that had zero electrical activity, that would be extra…”
- Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models ↗og_kalu · 2023-08-02 · 1 分