“Personally, I rate this paper”
No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
多模态模型零样本表现随概念频率呈对数线性增长
论文在 34 个模型和 5 个标准预训练数据集上研究下游概念在预训练数据中出现的频率如何影响多模态模型表现,发现所谓零样本泛化其实遵循样本效率很低的对数线性趋势:性能线性提高需要指数级更多的数据。该趋势在控制样本级相似度、以及在纯合成分布上测试时依然成立;作者还据此构建长尾测试集 Let it Wag!,各模型在上面普遍表现不佳。
评论者用它说明多模态和生成模型的进步需要指数级增长的数据而非算力,常作为「LLM 能力增长可能趋于平缓」的论据,也有人用它说明生成模型尚未真正做到零样本。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“researchers concluded that multi-modal models require exponentially more data for each extra "mode" being added to the model.”
“It appears that LLM capability growth is likely to plateau, although this is to be confirmed. No "Zero-Shot" Without Exponential Data”
“This is not about compute, but about data.”
“Research seems to suggest we need exponential training data volume increases to see meaningful performance gains: No "Zero-Shot" Without Exponential Data”
“Some research to the contrary [1] - tldr is that they didn't find evidence that generative models really do zero shot well at all yet”
看全部 9 条 →
“shows that these pre-trained few-shots learners exhibit a log-linear relationship with”
“EDIT: here's the paper No "Zero-Shot" Without Exponential Data”
“These diminishing returns are exponentially diminishing returns [1].”
- No "Zero-Shot" Without Exponential Data ↗kens · 2024-04-08 · 2 分
- No "Zero-Shot" Without Exponential Data ↗passwordoops · 2024-04-08 · 1 分
- No "Zero-Shot" Without Exponential Data ↗kmdupree · 2024-04-09 · 2 分
- No "Zero-Shot" Without Exponential Data ↗zerojames · 2024-05-09 · 187 分