按年被提到
20202022 · 1 次2026
视觉、语音与图像生成›ViT 与视觉新架构·arXiv:2012.12877 · cs.CV

Training data-efficient image transformers & distillation through attention

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou · 2020
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2022-03-10 · 最近 2023-11-28
2不同的人
2活跃月份
2讨论串
4,272被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论stevenwalton · 2022-03

“Lots of people transfer learn with transformers. ViT[0] originally did CIFAR with it. Then DeiT[1] introduced some knowledge transfer (note: their student is larger than the teacher). ViT pretrained on both ImageNet21k and JFT-300m. CCT ([1] from above) was focused on training from scratch. There's two paradigms to be aware of. ImageNet and pre-training…”

Transformers, originally designed to handle language, are taking on vision原评论 ↗
评论godelski · 2023-11

“> The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. Has it ever? I've read a lot of "old" papers and idk if there's ever been a strong theoretical framework. I'd say it is strongest today but not in main papers and those papers are getting rejected because of la…”

Simplifying Transformer Blocks原评论 ↗