mlsanityData to 7 Oct 2026

Vision Transformer with Deformable Attention

arXiv 2201.00520 ↗First posted 3 Jan 2022cs.CVcited by 293

Not on the map. Why?

Abstract (arXiv)

Transformers have recently shown superior performances on various vision tasks. The large, sometimes even global, receptive field endows Transformer models with higher representation power over their CNN counterparts. Nevertheless, simply enlarging receptive field also gives rise to several concerns. On the one hand, using dense attention e.g., in ViT, leads to excessive memory and computational cost, and features can be influenced by irrelevant parts which are beyond the region of interests. On the other hand, the sparse attention adopted in PVT or Swin Transformer is data agnostic and may limit the ability to model long range relations. To mitigate these issues, we propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant regions and capture more informative features. On this basis, we present Deformable Attention Transformer, a general backbone model with deformable attention for both image classification and dense prediction tasks. Extensive experiments show that our models achieve consistently improved results on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/DAT.

Key facts. Cited by 293 papers in the map (data through 7 Oct 2026). Builds on 50 of its 54 references that are in the map. First posted on arXiv 3 Jan 2022.

Builds on

50 of 54 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2016Layer Normalizationcited by 5,579
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2017Deformable Convolutional Networkscited by 1,481
  5. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  6. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  7. 2017Random Erasing Data Augmentationcited by 1,235
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  10. 2016Deep Networks with Stochastic Depthcited by 973
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  12. 2017Focal Loss for Dense Object Detectioncited by 749
  13. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  14. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 666
  15. 2019Panoptic Feature Pyramid Networkscited by 644
  16. 2021Transformer in Transformercited by 632
  17. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  18. 2021Early Convolutions Help Transformers See Bettercited by 395
  19. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  20. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  21. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  22. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  23. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  24. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  25. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  26. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  27. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  28. 20203D Object Detection with Pointformercited by 194
  29. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  30. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  31. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  32. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 99
  33. 2020Convolutional Networks with Dense Connectivitycited by 98
  34. 2021On the Integration of Self-Attention and Convolutioncited by 74
  35. 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
  36. 2021Vision Transformer with Progressive Samplingcited by 44
  37. 2021DPT: Deformable Patch-based Transformer for Visual Recognitioncited by 43
  38. 2021Visual Parser: Representing Part-whole Hierarchies with Transformerscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 37; 2023: 70; 2024: 70; 2025: 79; 2026: 37.

293 papers.

Sort:

By area:

  1. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  2. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  3. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  4. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  5. 2022Recurrent Video Restoration Transformer with Guided Deformable Attentioncited by 168
  6. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  7. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  8. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  9. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  10. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  11. 2022HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentationcited by 94
  12. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
38 more
  1. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  2. 2024HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Modelcited by 72
  3. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  4. 2024MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretrainingcited by 66
  5. 2023Adaptive Rotated Convolution for Rotated Object Detectioncited by 51
  6. 2023SpectFormer: Frequency and Attention is what you need in a Vision Transformercited by 48
  7. 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
  8. 2022Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentationcited by 46
  9. 2023Vision Transformer with Quadrangle Attentioncited by 46
  10. 2022VSA: Learning Varied-Size Window Attention in Vision Transformerscited by 43
  11. 2022Vision Transformer with Super Token Samplingcited by 43
  12. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 42
  13. 2023EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimationcited by 42
  14. 2022Glance and Focus Networks for Dynamic Visual Recognitioncited by 41
  15. 2023Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attentioncited by 38
  16. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  17. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  18. 2023Decoupling Common and Unique Representations for Multimodal Self-supervised Learningcited by 36
  19. 2022Contrastive Language-Image Pre-Training with Knowledge Graphscited by 34
  20. 2024Frequency-Adaptive Dilated Convolution for Semantic Segmentationcited by 34
  21. 2022Class-Aware Adversarial Transformers for Medical Image Segmentationcited by 33
  22. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  23. 2022Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectivescited by 32
  24. 2023Audio-aware Query-enhanced Transformer for Audio-Visual Segmentationcited by 32
  25. 2022Transformers in 3D Point Clouds: A Surveycited by 31
  26. 2024Breaking the Low-Rank Dilemma of Linear Attentioncited by 31
  27. 2023FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detectioncited by 30
  28. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 30
  29. 2023UniBEV: Multi-modal 3D Object Detection with Uniform BEV Encoders for Robustness against Missing Sensor Modalitiescited by 28
  30. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  31. 2023DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Predictioncited by 27
  32. 2025IDEA: Inverted Text with Cooperative Deformable Aggregation for Multi-modal Object Re-Identificationcited by 27
  33. 2023OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformercited by 26
  34. 2026Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Drivingcited by 26
  35. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  36. 2023Lightweight Vision Transformer with Bidirectional Interactioncited by 25
  37. 2023SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentationcited by 25
  38. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24

50 most cited of 293 All 293 (JSON)