mlsanityData to 7 Oct 2026

Co-Scale Conv-Attentional Image Transformers

arXiv 2104.06399 ↗First posted 13 Apr 2021cs.CV cs.LG cs.NEcited by 224

Not on the map. Why?

Abstract (arXiv)

In this paper, we present Co-scale conv-attentional image Transformers (CoaT), a Transformer-based image classifier equipped with co-scale and conv-attentional mechanisms. First, the co-scale mechanism maintains the integrity of Transformers' encoder branches at individual scales, while allowing representations learned at different scales to effectively communicate with each other; we design a series of serial and parallel blocks to realize the co-scale mechanism. Second, we devise a conv-attentional mechanism by realizing a relative position embedding formulation in the factorized attention module with an efficient convolution-like implementation. CoaT empowers image Transformers with enriched multi-scale and contextual modeling capabilities. On ImageNet, relatively small CoaT models attain superior classification results compared with similar-sized convolutional neural networks and image/vision Transformers. The effectiveness of CoaT's backbone is also illustrated on object detection and instance segmentation, demonstrating its applicability to downstream computer vision tasks.

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

42 of 57 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
30 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2017Deformable Convolutional Networkscited by 1,481
  13. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  14. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  15. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  16. 2017Random Erasing Data Augmentationcited by 1,235
  17. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  18. 2018Self-Attention with Relative Position Representationscited by 1,112
  19. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  20. 2016Deep Networks with Stochastic Depthcited by 973
  21. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  22. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  23. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  24. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  25. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  26. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  27. 2020Exploring Self-attention for Image Recognitioncited by 313
  28. 2019Local Relation Networks for Image Recognitioncited by 210
  29. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  30. 2021Involution: Inverting the Inherence of Convolution for Visual Recognitioncited by 99

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 74; 2023: 58; 2024: 28; 2025: 16; 2026: 7.

224 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  4. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  5. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  6. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  7. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  8. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  9. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  10. 2022How Do Vision Transformers Work?cited by 305
  11. 2022Vision Transformer with Deformable Attentioncited by 293
  12. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
38 more
  1. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  2. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  3. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  4. 2022Visual Attention Networkcited by 210
  5. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  6. 2022Focal Modulation Networkscited by 186
  7. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  8. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  9. 2022DaViT: Dual Attention Vision Transformerscited by 174
  10. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  11. 2021UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagerycited by 166
  12. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  13. 2022Efficient Frequency Domain-based Transformers for High-Quality Image Deblurringcited by 164
  14. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  15. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  16. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  17. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  18. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  19. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  20. 2021Vision Transformer for Small-Size Datasetscited by 122
  21. 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
  22. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  23. 2022Inception Transformercited by 99
  24. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  25. 2021Multi-Scale High-Resolution Vision Transformer for Semantic Segmentationcited by 92
  26. 2023CompletionFormer: Depth Completion with Convolutions and Vision Transformerscited by 92
  27. 2021Contextual Transformer Networks for Visual Recognitioncited by 90
  28. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  29. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  30. 2023MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazingcited by 81
  31. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  32. 2022Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learningcited by 78
  33. 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
  34. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  35. 2022Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarioscited by 65
  36. 2022Underwater Ranker: Learn Which Is Better and How to Be Bettercited by 65
  37. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  38. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64

50 most cited of 224 All 224 (JSON)