mlsanityData to 7 Oct 2026

SimMIM: A Simple Framework for Masked Image Modeling

arXiv 2111.09886 ↗First posted 18 Nov 2021cs.CVcited by 1,147

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents SimMIM, a simple framework for masked image modeling. We simplify recently proposed related approaches without special designs such as block-wise masking and tokenization via discrete VAE or clustering. To study what let the masked image modeling task learn good representations, we systematically study the major components in our framework, and find that simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a strong pre-text task; 2) predicting raw pixels of RGB values by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K by pre-training also on this dataset, surpassing previous best approach by +0.6%. When applied on a larger model of about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to facilitate the training of a 3B model (SwinV2-G), that by 40× less data than that in previous practice, we achieve the state-of-the-art on four representative vision benchmarks. The code and models will be publicly available at https://github.com/microsoft/SimMIM.

Key facts. Cited by 1,147 papers in the map (data through 7 Oct 2026). Builds on 50 of its 62 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

50 of 62 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2017Mask R-CNNcited by 7,259
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  10. 2017The Kinetics Human Action Video Datasetcited by 1,894
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  13. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  14. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  16. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  17. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  18. 2017Random Erasing Data Augmentationcited by 1,235
  19. 2016Colorful Image Colorizationcited by 1,113
  20. 2021Video Swin Transformercited by 1,090
  21. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  22. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  23. 2021Scaling Vision Transformerscited by 993
  24. 2016Deep Networks with Stochastic Depthcited by 973
  25. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  26. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  27. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  28. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  29. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  30. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  31. 2021Dynamic Neural Networks: A Surveycited by 405
  32. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  33. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  34. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  35. 2016Semantic Image Inpainting with Deep Generative Modelscited by 156
  36. 2020Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolationcited by 51
  37. 2019Selfie: Self-supervised Pretraining for Image Embeddingcited by 46
  38. 2020Parametric Instance Classification for Unsupervised Visual Feature Learningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 189; 2023: 343; 2024: 286; 2025: 202; 2026: 124.

1,147 papers.

Sort:

By area:

  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  8. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  9. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  10. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  11. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  12. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
38 more
  1. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  2. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  3. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  4. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  5. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  6. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  7. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  8. 2024Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCIcited by 311
  9. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  10. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  11. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  12. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  13. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  14. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  15. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  16. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  17. 2022Visual Prompting via Image Inpaintingcited by 248
  18. 2023A Cookbook of Self-Supervised Learningcited by 248
  19. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  20. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  21. 2022Test-Time Training with Masked Autoencoderscited by 208
  22. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  23. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  24. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  25. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  26. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 188
  27. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  28. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  29. 2022MIC: Masked Image Consistency for Context-Enhanced Domain Adaptationcited by 176
  30. 2022Masked World Models for Visual Controlcited by 169
  31. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  32. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 165
  33. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  34. 2022Advancing Plain Vision Transformer Towards Remote Sensing Foundation Modelcited by 149
  35. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  36. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  37. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
  38. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133

50 most cited of 1,147 All 1,147 (JSON)