mlsanityData to 14 Oct 2026

Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN

arXiv 2205.13943 ↗First posted 27 May 2022cs.CV cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Masked image modeling, an emerging self-supervised pre-training method, has shown impressive success across numerous downstream vision tasks with Vision transformers. Its underlying idea is simple: a portion of the input image is masked out and then reconstructed via a pre-text task. However, the working principle behind MIM is not well explained, and previous studies insist that MIM primarily works for the Transformer family but is incompatible with CNNs. In this work, we observe that MIM essentially teaches the model to learn better middle-order interactions among patches for more generalized feature extraction. We then propose an Architecture-Agnostic Masked Image Modeling framework (A2MIM), which is compatible with both Transformers and CNNs in a unified way. Extensive experiments on popular benchmarks show that A2MIM learns better representations without explicit design and endows the backbone model with the stronger capability to transfer to various downstream tasks.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 67 of its 93 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

67 of 93 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017Mask R-CNNcited by 7,277
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
38 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  2. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2022A ConvNet for the 2020scited by 3,579
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  13. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  14. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  15. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  16. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  17. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  18. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  19. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  20. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  21. 2019Contrastive Multiview Codingcited by 1,301
  22. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  23. 2017Random Erasing Data Augmentationcited by 1,239
  24. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  25. 2016Colorful Image Colorizationcited by 1,114
  26. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  27. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  28. 2016Deep Networks with Stochastic Depthcited by 979
  29. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  30. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  31. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  32. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  33. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  34. 2021Intriguing Properties of Vision Transformerscited by 446
  35. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  36. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  37. 2022How Do Vision Transformers Work?cited by 307
  38. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 12; 2024: 9; 2025: 4; 2026: 1.

31 papers.

Sort:

By area:

  1. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  2. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  3. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  4. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  5. 2023OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learningcited by 54
  6. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  7. 2023GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Trainingcited by 38
  8. 2024Masked Image Modeling: A Surveycited by 29
  9. 2023Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysiscited by 26
  10. 2023Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignalscited by 23
  11. 2022SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learnerscited by 21
  12. 2024A Survey on Mixup Augmentations and Beyondcited by 20
19 more
  1. 2023Adversarial AutoMixupcited by 16
  2. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  3. 2024SUMix: Mixup with Semantic and Uncertain Informationcited by 12
  4. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  5. 2023CMAE-V: Contrastive Masked Autoencoders for Video Action Recognitioncited by 11
  6. 2024Switch EMA: A Free Lunch for Better Flatness and Sharpnesscited by 9
  7. 2024Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretrainingcited by 7
  8. 2025Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Modelscited by 7
  9. 2022Transfer Learning on Electromyography (EMG) Tasks: Approaches and Beyondcited by 5
  10. 2023Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encodercited by 5
  11. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  12. 2024Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordingscited by 3
  13. 2024Self-Supervised Learning for Real-World Object Detection: a Surveycited by 2
  14. 2023Valid Information Guidance Network for Compressed Video Quality Enhancementcited by 1
  15. 2024D2SL: Decouple Defogging and Semantic Learning for Foggy Domain-Adaptive Segmentationcited by 1
  16. 2025MINR: Implicit Neural Representations with Masked Image Modellingcited by 1
  17. 2024Symmetric masking strategy enhances the performance of Masked Image Modelingcited by 0
  18. 2025CoMA: Complementary Masking and Hierarchical Dynamic Multi-Window Self-Attention in a Unified Pre-training Frameworkcited by 0
  19. 2026Learning from Semantic Dictionaries: Discriminative Codebook Contrastive Learning for Unified Visual Representation and Generationcited by 0