mlsanityData to 7 Oct 2026

Exploring Plain Vision Transformer Backbones for Object Detection

arXiv 2203.16527 ↗First posted 30 Mar 2022cs.CVcited by 723

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very few cross-window propagation blocks. With plain ViT backbones pre-trained as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the previous leading methods that were all based on hierarchical backbones, reaching up to 61.3 AP_box on the COCO dataset using only ImageNet-1K pre-training. We hope our study will draw attention to research on plain-backbone detectors. Code for ViTDet is available in Detectron2.

Key facts. Cited by 723 papers in the map (data through 7 Oct 2026). Builds on 51 of its 60 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

51 of 60 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  3. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  8. 2015Fast R-CNNcited by 2,930
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  13. 2018Group Normalizationcited by 1,514
  14. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  15. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  16. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  17. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  18. 2021Scaling Vision Transformerscited by 993
  19. 2021Multiscale Vision Transformerscited by 896
  20. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  21. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  22. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 719
  23. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  24. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  25. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  26. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  27. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  28. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  29. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  30. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  31. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  32. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  33. 2021Dynamic Head: Unifying Object Detection Heads with Attentionscited by 185
  34. 2019Scale-Aware Trident Networks for Object Detectioncited by 159
  35. 2021Probabilistic two-stage detectioncited by 130
  36. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 130
  37. 2021You Only Look One-level Featurecited by 116
  38. 2021CBNet: A Composite Backbone Network Architecture for Object Detectioncited by 77

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 72; 2023: 209; 2024: 188; 2025: 155; 2026: 99.

723 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,158
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  4. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  5. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  6. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  9. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  10. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  11. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  12. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
38 more
  1. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  2. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  3. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  4. 2023Segment Anything in High Qualitycited by 373
  5. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  6. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  7. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  8. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  9. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  10. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  11. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  12. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  13. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  14. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  15. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  16. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  17. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  18. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  19. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  20. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  21. 2023RTMPose: Real-Time Multi-Person Pose Estimation based on MMPosecited by 170
  22. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  23. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  24. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  25. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  26. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  27. 2022A Unified Sequence Interface for Vision Taskscited by 152
  28. 2023BiFormer: Vision Transformer with Bi-Level Routing Attentioncited by 150
  29. 2022DETRs with Hybrid Matchingcited by 149
  30. 2022Advancing Plain Vision Transformer Towards Remote Sensing Foundation Modelcited by 149
  31. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  32. 2023PACO: Parts and Attributes of Common Objectscited by 146
  33. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  34. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
  35. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  36. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  37. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 128
  38. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119

50 most cited of 723 All 723 (JSON)