按年被提到
20222026 · 2 次2026
视觉、语音与图像生成›视觉语言与自监督表征·arXiv:2203.05482 · cs.LG

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-12-31 · 最近 2026-06-14
2不同的人
3活跃月份
3讨论串
350被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github.com/mlfoundations/model-soups.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论magicalhippo · 2024-12

“This is probably a dumb idea, but I'll air it anyway. I just stumbled upon the model soup[1] paper where, as I understand it, they average weights of fine-tuned models and get an model that performs better. They have a more involved algorithm but even the uniform soup (simple weight average) seems to perform well. In your paper you mention that especial…”

Beyond Gradient Averaging in Parallel Optimization原评论 ↗
评论magicalhippo · 2026-03

“Maybe some newer references are better, but my mind went to the Model Soups paper[1]: The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of …”

NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute原评论 ↗
评论woadwarrior01 · 2026-06

“It's is a well known idea[1], although it's still surprising that something as simple, even works. [1]: https://arxiv.org/abs/2203.05482”

Rio de Janeiro's "homegrown" LLM appears to be a merge of an existing model原评论 ↗