“you can even use them as discriminative models for images”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
把图像切成 16x16 图块直接交给纯 Transformer 做分类
论文表明,视觉任务并不需要依赖 CNN:把图像切成图块序列,直接用纯 Transformer(Vision Transformer,ViT)就能在图像分类上表现很好。在大规模数据上预训练、再迁移到 ImageNet、CIFAR-100、VTAB 等中小型基准时,ViT 取得了与最先进卷积网络相比的优异结果,而训练所需算力明显更少。
评论者把它当作“图像切成图块当 token、无需先用 CNN 提取特征”的出处来引用,用来解释图像在 Transformer 里的 token 到底是什么。
学习路径
读这篇之前,按顺序读书架上的 5 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径补齐 ViT 的三块前置:Transformer 架构、大规模预训练的范式,以及图像侧的大数据迁移。
- 1
- 2
- 3
ViT 的分类 token 与先预训练再微调的流程直接来自 BERT
读看 [CLS] 标记的用法与预训练加微调的范式
- 4
说明数据量增大对视觉任务的影响,ViT 同样依赖大数据集
读JFT-300M 上数据规模与性能的关系,看大数据预训练的收益
- 5
大数据预训练加迁移的卷积基线,ViT 的主要对比对象
读BiT 的预训练与迁移流程,以及它在各基准上的结果
- 6An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
目标论文:纯 Transformer 处理图像块序列
读图像切块嵌入的做法,以及数据规模如何决定它是否胜过 CNN
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“recent studies [2] show that they can be applied to computer vision tasks.”
“ViT[0] originally did CIFAR with it.”
“I don't quite remember whether it was first used in Vit paper[1], but it's a fairly straight forward idea.”
“Very interesting. I didn't think of connecting these two dots.”
“I will refer you to the Vision Transformer: An Image is Worth 16x16 Words”
看全部 15 条 →
“and the fact that you didn't need CNNs to extract features first didn't really come into light till this paper - An Image is Worth 16x16 Words”
“Vision Transformers don't use CNNs to extract anything first..”
“ViT (ICLR 21) and DeiT (ICML 21)[4,5], and even ResNet (CVPR 16)[6] and tons of ReLU papers we can list.”
“Each image is about 258 tokens because a token in image transformer terms is literally a patch of the image.”
“I literally spend most of my days in ViTs, but thanks for the link.”
“Just tried it on this paper: An Image is Worth 16x16 Words”
“(2020) An Image is Worth 16x16 Words: an image is worth 16x16 words transformers for image recognition at scale”
“Although it looks like that was just ImageNet so maybe”
“Vision Transformers, which apply it to images, came three years later in 2020: An Image is Worth 16x16 Words”
- An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale ↗doener · 2022-04-01 · 8 分