mlsanityData to 7 Oct 2026

How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders

arXiv 2210.08344 ↗First posted 15 Oct 2022cs.LG cs.CVcited by 86

Not on the map. Why?

Abstract (arXiv)

Masked Autoencoders (MAE) based on a reconstruction task have risen to be a promising paradigm for self-supervised learning (SSL) and achieve state-of-the-art performance across different benchmark datasets. However, despite its impressive empirical success, there is still limited theoretical understanding of it. In this paper, we propose a theoretical understanding of how masking matters for MAE to learn meaningful features. We establish a close connection between MAE and contrastive learning, which shows that MAE implicit aligns the mask-induced positive pairs. Built upon this connection, we develop the first downstream guarantees for MAE methods, and analyze the effect of mask ratio. Besides, as a result of the implicit alignment, we also point out the dimensional collapse issue of MAE, and propose a Uniformity-enhanced MAE (U-MAE) loss that can effectively address this issue and bring significant improvements on real-world datasets, including CIFAR-10, ImageNet-100, and ImageNet-1K. Code is available at (https://github.com/zhangq327/U-MAE).

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 15 Oct 2022.

Builds on

24 of 36 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
12 more
  1. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  2. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  3. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  4. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  5. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  6. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  7. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  8. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  9. 2022How to Understand Masked Autoencoderscited by 37
  10. 2020Implicit Rank-Minimizing Autoencodercited by 35
  11. 2021Residual Relaxation for Multi-view Representation Learningcited by 30
  12. 2022Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 28; 2025: 22; 2026: 17.

86 papers.

Sort:

By area:

  1. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  2. 2025Diffuse and Disperse: Image Generation with Representation Regularizationcited by 67
  3. 2024Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Trainingcited by 57
  4. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  5. 2023Simple and Asymmetric Graph Contrastive Learning without Augmentationscited by 37
  6. 2024Masked Image Modeling: A Surveycited by 28
  7. 2023CL-MAE: Curriculum-Learned Masked Autoencoderscited by 21
  8. 2025HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Seriescited by 19
  9. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  10. 2024Non-negative Contrastive Learningcited by 18
  11. 2024UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learningcited by 18
  12. 2023Information Flow in Self-Supervised Learningcited by 16
38 more
  1. 2023Adversarial Examples Are Not Real Featurescited by 16
  2. 2024One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representationcited by 16
  3. 2023Unlocking the diagnostic potential of electrocardiograms through information transfer from cardiac magnetic resonance imagingcited by 14
  4. 2023Identifiable Contrastive Learning with Automatic Feature Importance Discoverycited by 14
  5. 2024Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-trainingcited by 14
  6. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  7. 2024ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopycited by 13
  8. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 12
  9. 2024LDReg: Local Dimensionality Regularized Self-Supervised Learningcited by 11
  10. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  11. 2023RARE: Robust Masked Graph Autoencodercited by 10
  12. 2023Understanding Masked Autoencoders From a Local Contrastive Perspectivecited by 10
  13. 2023Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusioncited by 10
  14. 2024Emerging Property of Masked Token for Effective Pre-trainingcited by 10
  15. 2024Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Imagescited by 9
  16. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  17. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  18. 2024On the Generalization Ability of Unsupervised Pretrainingcited by 8
  19. 2024ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoderscited by 8
  20. 2024Connect Later: Improving Fine-tuning for Robustness with Targeted Augmentationscited by 7
  21. 2024On the Role of Discrete Tokenization in Visual Representation Learningcited by 7
  22. 2024MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentationcited by 7
  23. 2024Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognitioncited by 7
  24. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 7
  25. 2023Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learningcited by 6
  26. 2024Beyond [cls]: Exploring the true potential of Masked Image Modeling representationscited by 6
  27. 2023Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encodercited by 5
  28. 2024Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyondcited by 5
  29. 2025An Augmentation Overlap Theory of Contrastive Learningcited by 5
  30. 2023On Separate Normalization in Self-supervised Transformerscited by 4
  31. 2025Insulin Resistance Prediction From Wearables and Routine Blood Biomarkerscited by 4
  32. 2025Discrepancy-Aware Graph Mask Auto-Encodercited by 4
  33. 2026Sapiens2cited by 4
  34. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  35. 2025Learning Mask Invariant Mutual Information for Masked Image Modelingcited by 3
  36. 2025InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signalscited by 3
  37. 2025Self-Guided Masked Autoencodercited by 3
  38. 2023Remote Sensing Scene Classification with Masked Image Modeling (MIM)cited by 2

50 most cited of 86 All 86 (JSON)