mlsanityData to 7 Oct 2026

CAT: Cross Attention in Vision Transformer

arXiv 2106.05786 ↗First posted 10 Jun 2021cs.CV cs.AIcited by 92

Not on the map. Why?

Abstract (arXiv)

Since Transformer has found widespread use in NLP, the potential of Transformer in CV has been realized and has inspired many new approaches. However, the computation required for replacing word tokens with image patches for Transformer after the tokenization of the image is vast(e.g., ViT), which bottlenecks model training and inference. In this paper, we propose a new attention mechanism in Transformer termed Cross Attention, which alternates attention inner the image patch instead of the whole image to capture local information and apply attention between image patches which are divided from single-channel feature maps capture global information. Both operations have less computation than standard self-attention in Transformer. By alternately applying attention inner patch and between patches, we implement cross attention to maintain the performance with lower computational cost and build a hierarchical network called Cross Attention Transformer(CAT) for other vision tasks. Our base model achieves state-of-the-arts on ImageNet-1K, and improves the performance of other methods on COCO and ADE20K, illustrating that our network has the potential to serve as general backbones. The code and models are available at urlhttps://github.com/linhezheng19/CAT.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 78 of its 90 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

78 of 90 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2016Layer Normalizationcited by 5,579
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  12. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  13. 2015Fast R-CNNcited by 2,930
  14. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  15. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  16. 2018YOLOv3: An Incremental Improvementcited by 2,652
  17. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  18. 2016Pyramid Scene Parsing Networkcited by 2,472
  19. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  20. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  21. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  22. 2019Searching for MobileNetV3cited by 1,984
  23. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  24. 2017Decoupled Weight Decay Regularizationcited by 1,722
  25. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  26. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  27. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  28. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  29. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  30. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  31. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  32. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  33. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  34. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  35. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  36. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  37. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  38. 2018Dual Attention Network for Scene Segmentationcited by 1,001

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 13; 2024: 16; 2025: 38; 2026: 15.

92 papers.

Sort:

By area:

  1. 2022Vision Transformers for Single Image Dehazingcited by 255
  2. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  3. 2022SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detectioncited by 59
  4. 20233Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognitioncited by 40
  5. 2022Cascade Transformers for End-to-End Person Searchcited by 31
  6. 2022Beyond Fixation: Dynamic Window Visual Transformercited by 28
  7. 2021BiTr-Unet: a CNN-Transformer Combined Network for MRI Brain Tumor Segmentationcited by 19
  8. 2023CrossFormer++: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 18
  9. 2024HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Accelerationcited by 16
  10. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  11. 20233DSGrasp: 3D Shape-Completion for Robotic Graspcited by 13
  12. 2025Multi-Modal Multi-Behavior Sequential Recommendation with Conditional Diffusion-Based Feature Denoisingcited by 13
38 more
  1. 2024Motion meets Attention: Video Motion Promptscited by 12
  2. 2023Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inferencecited by 11
  3. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  4. 2025Erasing Concepts, Steering Generations: A Comprehensive Survey of Concept Suppressioncited by 10
  5. 2022Learning Cross-Image Object Semantic Relation in Transformer for Few-Shot Fine-Grained Image Classificationcited by 9
  6. 2023Towards Simultaneous Segmentation of Liver Tumors and Intrahepatic Vessels via Cross-attention Mechanismcited by 8
  7. 2023ESTformer: Transformer utilising spatiotemporal dependencies for electroencephalogram super-resolutioncited by 8
  8. 2025Reasoning to Edit: Hypothetical Instruction-Based Image Editing with Visual Reasoningcited by 8
  9. 2024ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Imagescited by 7
  10. 2024Adapting Vision Foundation Models for Robust Cloud Segmentation in Remote Sensing Imagescited by 7
  11. 2025BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Modelcited by 7
  12. 2025Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakescited by 7
  13. 2023Alternative Telescopic Displacement: An Efficient Multimodal Alignment Methodcited by 6
  14. 2024Refusing Safe Prompts for Multi-modal Large Language Modelscited by 6
  15. 2023Factorization Vision Transformer: Modeling Long Range Dependency with Local Window Costcited by 5
  16. 2024Zoom and Shift are All You Needcited by 5
  17. 2024Complementary Advantages: Exploiting Cross-Field Frequency Correlation for NIR-Assisted Image Denoisingcited by 5
  18. 2025BeamLLM: Vision-Empowered mmWave Beam Prediction with Large Language Modelscited by 5
  19. 2023You Only Train Once: A Unified Framework for Both Full-Reference and No-Reference Image Quality Assessmentcited by 4
  20. 2025FDG-Diff: Frequency-Domain-Guided Diffusion Framework for Compressed Hazy Image Restorationcited by 4
  21. 2021Hformer: Hybrid CNN-Transformer for Fringe Order Prediction in Phase Unwrapping of Fringe Projectioncited by 3
  22. 2022Dense-TNT: Efficient Vehicle Type Classification Neural Network Using Satellite Imagerycited by 3
  23. 2023DUFormer: Solving Power Line Detection Task in Aerial Images using Semantic Segmentationcited by 3
  24. 2023Vision Backbone Enhancement via Multi-Stage Cross-Scale Attentioncited by 3
  25. 2024ReViT: Enhancing Vision Transformers Feature Diversity with Attention Residual Connectionscited by 3
  26. 2025Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratingscited by 3
  27. 2025SKALD: Learning-Based Shot Assembly for Coherent Multi-Shot Video Creationcited by 3
  28. 2025Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Modelscited by 3
  29. 2025Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglementcited by 3
  30. 2024Probing Fine-Grained Action Understanding and Cross-View Generalization of Foundation Modelscited by 2
  31. 2024LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generationcited by 2
  32. 2025Learning Joint ID-Textual Representation for ID-Preserving Image Synthesiscited by 2
  33. 2025SoilNet: A Multimodal Multitask Model for Hierarchical Classification of Soil Horizonscited by 2
  34. 2025Midway Network: Learning Representations for Recognition and Motion from Latent Dynamicscited by 2
  35. 2025The SAM2-to-SAM3 Gap in the Segment Anything Model Family: Why Prompt-Based Expertise Fails in Concept-Driven Image Segmentationcited by 2
  36. 2025Sample from What You See: Visuomotor Policy Learning via Diffusion Bridge with Observation-Embedded Stochastic Differential Equationcited by 2
  37. 2026STENet: Superpixel Token Enhancing Network for RGB-D Salient Object Detectioncited by 2
  38. 2023GeoViT: A Versatile Vision Transformer Architecture for Geospatial Image Analysiscited by 1

50 most cited of 92 All 92 (JSON)