mlsanityData to 7 Oct 2026

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

arXiv 2103.14899 ↗First posted 27 Mar 2021cs.CVcited by 806

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recently developed vision transformer (ViT) has achieved promising results on image classification compared to convolutional neural networks. Inspired by this, in this paper, we study how to learn multi-scale feature representations in transformer models for image classification. To this end, we propose a dual-branch transformer to combine image patches (i.e., tokens in a transformer) of different sizes to produce stronger image features. Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity and these tokens are then fused purely by attention multiple times to complement each other. Furthermore, to reduce computation, we develop a simple yet effective token fusion module based on cross attention, which uses a single token for each branch as a query to exchange information with other branches. Our proposed cross-attention only requires linear time for both computational and memory complexity instead of quadratic time otherwise. Extensive experiments demonstrate that our approach performs better than or on par with several concurrent works on vision transformer, in addition to efficient CNN models. For example, on the ImageNet1K dataset, with some architectural changes, our approach outperforms the recent DeiT by a large margin of 2% with a small to moderate increase in FLOPs and model parameters. Our source codes and models are available at urlhttps://github.com/IBM/CrossViT.

Key facts. Cited by 806 papers in the map (data through 7 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 27 Mar 2021.

Builds on

41 of 49 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
29 more
  1. 2017Non-local Neural Networkscited by 2,290
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2018SlowFast Networks for Video Recognitioncited by 1,862
  5. 2017Random Erasing Data Augmentationcited by 1,235
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2020Designing Network Design Spacescited by 954
  10. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  11. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  12. 2016Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurringcited by 722
  13. 2021Transformer in Transformercited by 632
  14. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 572
  15. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  16. 2019Selective Kernel Networkscited by 401
  17. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  18. 2019Attention Augmented Convolutional Networkscited by 363
  19. 2020Exploring Self-attention for Image Recognitioncited by 313
  20. 2019HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimationcited by 247
  21. 2016Using Deep Learning for Image-Based Plant Disease Detectioncited by 216
  22. 2019Local Relation Networks for Image Recognitioncited by 210
  23. 2019Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolutioncited by 140
  24. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  25. 2016A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detectioncited by 83
  26. 2019More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregationcited by 64
  27. 2015Multi-scale recognition with DAG-CNNscited by 23
  28. 2018Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognitioncited by 22
  29. 2021Centroid Transformers: Learning to Abstract with Attentioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 76; 2022: 168; 2023: 169; 2024: 183; 2025: 133; 2026: 77.

806 papers.

Sort:

By area:

  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2024VMamba: Visual State Space Modelcited by 955
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  4. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  5. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  6. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  7. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  8. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  9. 2022Multimodal Learning with Transformers: A Surveycited by 399
  10. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  11. 2022MAXIM: Multi-Axis MLP for Image Processingcited by 301
  12. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
38 more
  1. 2022Vision Transformers for Single Image Dehazingcited by 255
  2. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  3. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  4. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  5. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  6. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  7. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  8. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  9. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  10. 2022Focal Modulation Networkscited by 186
  11. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  12. 2022DaViT: Dual Attention Vision Transformerscited by 174
  13. 2021ASFormer: Transformer for Action Segmentationcited by 166
  14. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  15. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  16. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  17. 2021MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimationcited by 151
  18. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  19. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  20. 2022Multiview Transformers for Video Recognitioncited by 148
  21. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  22. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  23. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  24. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  25. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  26. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  27. 2021DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentationcited by 126
  28. 2021SwinTrack: A Simple and Strong Baseline for Transformer Trackingcited by 126
  29. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  30. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  31. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  32. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  33. 2021PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformercited by 118
  34. 2021A Survey of Visual Transformerscited by 117
  35. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  36. 2021Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformercited by 115
  37. 2024AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulationcited by 108
  38. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101

50 most cited of 806 All 806 (JSON)