mlsanityData to 14 Oct 2026

On Data Scaling in Masked Image Modeling

arXiv 2206.04664 ↗First posted 9 Jun 2022cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

An important goal of self-supervised learning is to enable model pre-training to benefit from almost unlimited data. However, one method that has recently become popular, namely masked image modeling (MIM), is suspected to be unable to benefit from larger data. In this work, we break this misconception through extensive experiments, with data scales ranging from 10% of ImageNet-1K to full ImageNet-22K, model sizes ranging from 49 million to 1 billion, and training lengths ranging from 125K iterations to 500K iterations. Our study reveals that: (i) Masked image modeling is also demanding on larger data. We observed that very large models got over-fitted with relatively small data; (ii) The length of training matters. Large models trained with masked image modeling can benefit from more data with longer training; (iii) The validation loss in pre-training is a good indicator to measure how well the model performs for fine-tuning on multiple tasks. This observation allows us to pre-evaluate pre-trained models in advance without having to make costly trial-and-error assessments of downstream tasks. We hope that our findings will advance the understanding of masked image modeling in terms of scaling ability.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

40 of 51 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017Mask R-CNNcited by 7,277
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
28 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  11. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  12. 2017Random Erasing Data Augmentationcited by 1,239
  13. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  14. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  15. 2021Video Swin Transformercited by 1,091
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  17. 2021Scaling Vision Transformerscited by 999
  18. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  19. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  20. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  21. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  22. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  23. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  24. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  25. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  26. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  27. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  28. 2021A Near-Optimal Algorithm for Debiasing Trained Machine Learning Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 14; 2024: 15; 2025: 10; 2026: 1.

46 papers.

Sort:

  1. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  2. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  3. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  4. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  5. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  6. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  7. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  8. 2022SVFormer: Semi-supervised Video Transformer for Action Recognitioncited by 68
  9. 2024Scaling Wearable Foundation Modelscited by 63
  10. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  11. 2023Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecturecited by 46
  12. 2023Equivariant Adaptation of Large Pretrained Modelscited by 41
34 more
  1. 2022ResFormer: Scaling ViTs with Multi-Resolution Trainingcited by 34
  2. 2024Masked Image Modeling: A Surveycited by 29
  3. 2023MAPSeg: Unified Unsupervised Domain Adaptation for Heterogeneous Medical Image Segmentation Based on 3D Masked Autoencoding and Pseudo-Labelingcited by 25
  4. 2024SARATR-X: Toward Building A Foundation Model for SAR Target Recognitioncited by 25
  5. 2025TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observationcited by 25
  6. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Modelscited by 24
  7. 2024MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representationscited by 19
  8. 2023DeepMIM: Deep Supervision for Masked Image Modelingcited by 16
  9. 2024Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learningcited by 16
  10. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  11. 2023Contrastive Tuning: A Little Help to Make Masked Autoencoders Forgetcited by 15
  12. 2025Towards a general-purpose foundation model for fMRI analysiscited by 15
  13. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  14. 2024Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervisioncited by 13
  15. 2024OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagerycited by 11
  16. 2024AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-maskingcited by 10
  17. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  18. 2022Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methodscited by 8
  19. 2023Understanding Self-Supervised Pretraining with Part-Aware Representation Learningcited by 7
  20. 2024On Pretraining Data Diversity for Self-Supervised Learningcited by 7
  21. 2025Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registrationcited by 7
  22. 2024GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small Datasetscited by 6
  23. 2023Delving Deeper into Data Scaling in Masked Image Modelingcited by 5
  24. 2025Scaling Vision Transformers for Functional MRI with Flat Mapscited by 5
  25. 2025A Genealogy of Foundation Models in Remote Sensingcited by 4
  26. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  27. 2024AstroMAE: Redshift Prediction Using a Masked Autoencoder with a Novel Fine-Tuning Architecturecited by 3
  28. 2025Understanding and Enhancing Mask-Based Pretraining towards Universal Representationscited by 3
  29. 2024Capturing Temporal Components for Time Series Classificationcited by 2
  30. 2024Generating Automatically Print/Scan Textures for Morphing Attack Detection Applicationscited by 2
  31. 2025On Model and Data Scaling for Skeleton-based Self-Supervised Gait Recognitioncited by 2
  32. 2025Progressive Scaling Visual Object Trackingcited by 1
  33. 2025SimMIL: A Universal Weakly Supervised Pre-Training Framework for Multi-Instance Learning in Whole Slide Pathology Imagescited by 0
  34. 2026DIME: Scaling Facial Representation Learning via Differential Masked Autoencodingcited by 0