“A few more interesting papers not mentioned in the article:”
Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models
上下文学习能力主要取决于预训练数据覆盖,域外外推会失败
在受控设置下,用 (x, f(x)) 序列而非自然语言训练 Transformer,研究其如何在预训练数据混合的多个任务族之间做上下文识别与学习。结果显示任务族在预训练数据中有充分覆盖时,Transformer 表现出接近最优的无监督模型选择能力;但遇到域外任务或函数时,即使是简单外推也会出现多种失败模式。作者据此认为上下文学习能力与预训练数据覆盖的关系,比与基本泛化归纳偏置的关系更紧密。
HN 上的人反复引用它作为经验证据,支持「Transformer 难以超出训练数据分布泛化、LLM 不会推理」的论点。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Google Deep Mind just demonstrated that transformers don't generalize well outside their training set.”
“as LLMs can't extrapolate beyond the patterns in their training data (according to a paper from DeepMind last month):”
“as LLMs can't extrapolate beyond the patterns in their training data (according to a paper from DeepMind last month):”
“If after reading papers below that show empirically”
“Here, a few papers that show”
看全部 7 条 →
“Here are papers showing that these models can't reason:”
- Pretraining Data Mixtures Enable Narrow Model Selection Capabil. In Transformers ↗YeGoblynQueenne · 2023-11-06 · 1 分
- Evidence that transformers cannot generalize beyond their training data ↗agomez314 · 2023-11-06 · 3 分
- Transformers struggle with generalizing tasks beyond pre-training data ↗hislaziness · 2023-11-07 · 36 分