A Theoretical Framework for Masked Pretraining (MPT)
Not on the map. Why?
Abstract (arXiv)
Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various domains and achieves remarkable performance in multiple downstream tasks. However, the theoretical understanding of the working mechanism behind MPT is still limited. In this paper, we introduce a new theoretical framework to analyze MPT and understand the crucial role of masking in extracting meaningful representations. We establish theoretical connections between MPT and another popular self-supervised paradigm: contrastive learning. We prove that the masking technique implicitly creates positive pairs that are semantically similar and the reconstruction loss pulls them together in the feature space. Besides, as a result of the implicit alignment, we point out the dimensional collapse issue of MPT and propose a Uniformity-enhanced MPT (U-MPT) loss that can effectively address this issue and bring significant improvements in downstream tasks including linear evaluation, cross-dataset fine-tuning and out-of-distribution generalization on real-world data sets. Furthermore, we establish downstream guarantees of U-MPT and theoretically analyze the influence of masking strategies. Based on the theoretical analysis, we propose a new masking strategy which enhances the downstream performance of MPT and explains current improvements of masking strategies with our theoretical perspective.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 6 Sep 2026.
Builds on
48 of 60 references.
References · 48
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2017Neural Discrete Representation Learningcited by 4,712
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
36 more
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2019Natural Adversarial Examplescited by 1,408
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
- 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
- 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
- 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
- 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
- 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
- 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
- 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
- 2023Hard Patches Mining for Masked Image Modelingcited by 57
- 2021Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translationcited by 41
- 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
- 2022How to Understand Masked Autoencoderscited by 38
- 2020Implicit Rank-Minimizing Autoencodercited by 35
- 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
- 2021Residual Relaxation for Multi-view Representation Learningcited by 30
- 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
- 2023A Message Passing Perspective on Learning Dynamics of Contrastive Learningcited by 22
- 2023Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representationscited by 18
- 2023Rethinking Weak Supervision in Helping Contrastive Learningcited by 18
- 2024Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustnesscited by 13
- 2025Projection Head is Secretly an Information Bottleneckcited by 11
- 2023Representation Deficiency in Masked Language Modelingcited by 10
- 2024Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretrainingcited by 7
- 2024On the Role of Discrete Tokenization in Visual Representation Learningcited by 7
- 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelinescited by 6
- 2025An Augmentation Overlap Theory of Contrastive Learningcited by 5
- 2025An Augmentation-Aware Theory for Self-Supervised Contrastive Learningcited by 3
- 2025Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspectivecited by 2
Built on by
Not cited yet.