mlsanityData to 7 Oct 2026

Benchmarking Detection Transfer Learning with Vision Transformers

arXiv 2111.11429 ↗First posted 22 Nov 2021cs.CVcited by 130

Not on the map. Why?

Abstract (arXiv)

Object detection is a central downstream task used to test if pre-trained network parameters confer benefits, such as improved accuracy or training speed. The complexity of object detection methods can make this benchmarking non-trivial when new architectures, such as Vision Transformer (ViT) models, arrive. These difficulties (e.g., architectural incompatibility, slow training, high memory consumption, unknown training formulae, etc.) have prevented recent studies from benchmarking detection transfer learning with standard ViT models. In this paper, we present training techniques that overcome these challenges, enabling the use of standard ViT models as the backbone of Mask R-CNN. These tools facilitate the primary goal of our study: we compare five ViT initializations, including recent state-of-the-art self-supervised learning methods, supervised initialization, and a strong random initialization baseline. Our results show that recent masking-based unsupervised learning methods may, for the first time, provide convincing transfer learning improvements on COCO, increasing box AP up to 4% (absolute) over supervised and prior self-supervised pre-training methods. Moreover, these masking-based initializations scale better, with the improvement growing as model size increases.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

33 of 38 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
21 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2018Group Normalizationcited by 1,514
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  11. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  12. 2016Deep Networks with Stochastic Depthcited by 973
  13. 2021Multiscale Vision Transformerscited by 896
  14. 2021Going deeper with Image Transformerscited by 759
  15. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  16. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  17. 2018Rethinking ImageNet Pre-trainingcited by 376
  18. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  19. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  20. 2020TIDE: A General Toolbox for Identifying Object Detection Errorscited by 113
  21. 2020Toward Transformer-Based Object Detectioncited by 84

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 39; 2023: 37; 2024: 33; 2025: 14; 2026: 6.

130 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022Visual Prompt Tuningcited by 1,674
  3. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  4. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  5. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  6. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  7. 2023Universal Guidance for Diffusion Modelscited by 426
  8. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  9. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  10. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  11. 2022A Unified Sequence Interface for Vision Taskscited by 152
  12. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
38 more
  1. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  2. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  3. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  4. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  5. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  6. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 65
  7. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  8. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  9. 2023Hard Patches Mining for Masked Image Modelingcited by 56
  10. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  11. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  12. 2022MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformerscited by 46
  13. 2022Masked Image Modeling with Denoising Contrastcited by 45
  14. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  15. 2023IML-ViT: Benchmarking Image Manipulation Localization by Vision Transformercited by 42
  16. 2024LW-DETR: A Transformer Replacement to YOLO for Real-Time Detectioncited by 36
  17. 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34
  18. 2023Visual Tuningcited by 34
  19. 2022Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detectioncited by 33
  20. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  21. 2024An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixelscited by 32
  22. 2022A Closer Look at Self-Supervised Lightweight Vision Transformerscited by 31
  23. 2022mc-BEiT: Multi-choice Discretization for Image BERT Pre-trainingcited by 30
  24. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  25. 2022Towards Efficient 3D Object Detection with Knowledge Distillationcited by 29
  26. 2023Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matchingcited by 29
  27. 2022AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localizationcited by 28
  28. 2023Controllable Chest X-Ray Report Generation from Longitudinal Representationscited by 23
  29. 2024Playing to Vision Foundation Model's Strengths in Stereo Matchingcited by 23
  30. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  31. 2022Exploring Long-Sequence Masked Autoencoderscited by 22
  32. 2023Learning Expressive Prompting With Residuals for Vision Transformerscited by 21
  33. 2023Improving Masked Autoencoders by Learning Where to Maskcited by 20
  34. 2023SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channelscited by 20
  35. 2024PTM-VQA: Efficient Video Quality Assessment Leveraging Diverse PreTrained Models from the Wildcited by 17
  36. 2024Plancraft: an evaluation dataset for planning with LLM agentscited by 16
  37. 2023Contrastive Tuning: A Little Help to Make Masked Autoencoders Forgetcited by 15
  38. 2022S2WAT: Image Style Transfer via Hierarchical Vision Transformer using Strips Window Attentioncited by 14

50 most cited of 130 All 130 (JSON)