mlsanityData to 13 Oct 2026

XCiT: Cross-Covariance Image Transformers

arXiv 2106.09681 ↗First posted 17 Jun 2021cs.CV cs.LGcited by 337

Not on the map. Why?

Abstract (arXiv)

Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

Key facts. Cited by 337 papers in the map (data through 13 Oct 2026). Builds on 69 of its 86 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

69 of 86 references.

References · 69

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  6. 2020End-to-End Object Detection with Transformerscited by 7,396
  7. 2017Mask R-CNNcited by 7,266
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,426
  10. 2016Layer Normalizationcited by 5,624
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,027
  12. 2017Squeeze-and-Excitation Networkscited by 4,772
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  2. 2020Longformer: The Long-Document Transformercited by 3,393
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,544
  4. 2017Non-local Neural Networkscited by 2,291
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,266
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,961
  8. 2021Vision Transformers for Dense Predictioncited by 1,795
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  10. 2021ViViT: A Video Vision Transformercited by 1,615
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,614
  12. 2020Rethinking Attention with Performerscited by 1,594
  13. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  14. 2018Group Normalizationcited by 1,518
  15. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,510
  16. 2020Linformer: Self-Attention with Linear Complexitycited by 1,469
  17. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,442
  18. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,383
  19. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  20. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,268
  21. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  22. 2021Perceiver: General Perception with Iterative Attentioncited by 1,094
  23. 2016Deep Networks with Stochastic Depthcited by 977
  24. 2020Designing Network Design Spacescited by 956
  25. 2021Multiscale Vision Transformerscited by 896
  26. 2018Image Transformercited by 771
  27. 2021Going deeper with Image Transformerscited by 762
  28. 2019Panoptic Feature Pyramid Networkscited by 644
  29. 2021Transformer in Transformercited by 633
  30. 2017Fine-tuning CNN Image Retrieval with No Human Annotationcited by 498
  31. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  32. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  33. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 469
  34. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  35. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  36. 2019Axial Attention in Multidimensional Transformerscited by 356
  37. 2021FNet: Mixing Tokens with Fourier Transformscited by 346
  38. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 28; 2022: 95; 2023: 88; 2024: 69; 2025: 27; 2026: 30.

337 papers.

Sort:

By area:

  1. 2024VMamba: Visual State Space Modelcited by 963
  2. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  3. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  4. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  5. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 517
  6. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 482
  7. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  8. 2022DeiT III: Revenge of the ViTcited by 417
  9. 2022Vision Transformer with Deformable Attentioncited by 294
  10. 2022Neighborhood Attention Transformercited by 238
  11. 2023Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancementcited by 229
  12. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
38 more
  1. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 223
  2. 2022Visual Attention Networkcited by 210
  3. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  4. 2022DaViT: Dual Attention Vision Transformerscited by 174
  5. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 166
  6. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  7. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  8. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  9. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  10. 2022Understanding The Robustness in Vision Transformerscited by 144
  11. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  12. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  13. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 130
  14. 2023Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networkscited by 130
  15. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  16. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  17. 2023Dual Aggregation Transformer for Image Super-Resolutioncited by 124
  18. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  19. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  20. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  21. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  22. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  23. 2021Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstructioncited by 110
  24. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 107
  25. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  26. 2022Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimationcited by 100
  27. 2022Global Context Vision Transformerscited by 98
  28. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 98
  29. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95
  30. 2023Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Modelscited by 92
  31. 2021Contextual Transformer Networks for Visual Recognitioncited by 90
  32. 2022Three things everyone should know about Vision Transformerscited by 90
  33. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  34. 2021TransMix: Attend to Mix for Vision Transformerscited by 79
  35. 2023Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Modelscited by 79
  36. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  37. 2022MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstructioncited by 72
  38. 2022Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstructioncited by 67

50 most cited of 337 All 337 (JSON)