mlsanityData to 7 Oct 2026

CoAtNet: Marrying Convolution and Attention for All Data Sizes

arXiv 2106.04803 ↗First posted 9 Jun 2021cs.CV cs.LGcited by 566

Not on the map. Why?

Abstract (arXiv)

Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced "coat" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.

Key facts. Cited by 566 papers in the map (data through 7 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

46 of 55 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
34 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,763
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2014Going Deeper with Convolutionscited by 4,231
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  13. 2018Self-Attention with Relative Position Representationscited by 1,112
  14. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  15. 2021Transformers in Vision: A Surveycited by 1,009
  16. 2021Scaling Vision Transformerscited by 993
  17. 2016Deep Networks with Stochastic Depthcited by 973
  18. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  19. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  20. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  21. 2021Going deeper with Image Transformerscited by 759
  22. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  23. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  24. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  25. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  26. 2019Attention Augmented Convolutional Networkscited by 363
  27. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  28. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  29. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  30. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  31. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  32. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  33. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  34. 2020A Data and Compute Efficient Design for Limited-Resources Deep Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 44; 2022: 177; 2023: 131; 2024: 109; 2025: 66; 2026: 39.

566 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  5. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  6. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  7. 2024VMamba: Visual State Space Modelcited by 955
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  12. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
38 more
  1. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  4. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  5. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  6. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  7. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  10. 2022How Do Vision Transformers Work?cited by 305
  11. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  12. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  13. 2022Vision Transformers for Single Image Dehazingcited by 255
  14. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  15. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  16. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  17. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  18. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  19. 2022Visual Attention Networkcited by 210
  20. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  21. 2022Patches Are All You Need?cited by 201
  22. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  23. 2023Teaching CLIP to Count to Tencited by 194
  24. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  25. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  26. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  27. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  28. 2022DaViT: Dual Attention Vision Transformerscited by 174
  29. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  30. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  31. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  32. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  33. 2022Advancing Plain Vision Transformer Towards Remote Sensing Foundation Modelcited by 149
  34. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  35. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  36. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  37. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  38. 2022MetaFormer Baselines for Visioncited by 131

50 most cited of 566 All 566 (JSON)