按年被提到
20232023 · 2 次2026
视觉、语音与图像生成›ViT 与视觉新架构·arXiv:2302.05442 · cs.CV

Scaling Vision Transformers to 22 Billion Parameters

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer 等 42 人 · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-07-24 · 最近 2023-08-03
2不同的人
2活跃月份
2讨论串
140被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for "LLM-like" scaling in vision, and provides key steps towards getting there.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论alevskaya · 2023-07

“Yeah we used to use this in our older models years ago... I don't recall the details exactly, but I don't think it ever did very much. I certainly don't think it will help at all with stability. Things like Q/K layernorm are better tricks for softmax stability when scaling: https://arxiv.org/pdf/2302.05442.pdf”

Transformer Attention is off by one原评论 ↗
评论cs702 · 2023-08

“Keep in mind, even though this is the largest geospatial model, it's still a tiny model, with only 100M parameters. I'd be excited to see what a more substantial, state-of-the-art model could do with geospatial data. Say, a model based on something like ViT-22B, with 22 billion parameters: https://arxiv.org/abs/2302.05442.”

IBM and NASA Open Source Largest Geospatial AI Foundation Model on Hugging Face原评论 ↗