# Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data

arXiv: https://arxiv.org/abs/2601.20072 · HTML: https://mlsanity.com/paper/2601.20072/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2003.03773/) (2020)
- [SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders](https://mlsanity.com/paper/2206.10207/) (2022)
- [Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning](https://mlsanity.com/paper/1909.01804/) (2019)
- [SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classification](https://mlsanity.com/paper/2103.16725/) (2021)
- [Semi-supervised Vision Transformers at Scale](https://mlsanity.com/paper/2208.05688/) (2022)
- [Co-training with High-Confidence Pseudo Labels for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2301.04465/) (2023)
- [Semi-Supervised Vision Transformers](https://mlsanity.com/paper/2111.11067/) (2021)
- [SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learners](https://mlsanity.com/paper/2205.14540/) (2022)
- [Semi-Supervised Learning with Multi-Head Co-Training](https://mlsanity.com/paper/2107.04795/) (2021)
- [InPL: Pseudo-labeling the Inliers First for Imbalanced Semi-supervised Learning](https://mlsanity.com/paper/2303.07269/) (2023)
- [RegMixMatch: Optimizing Mixup Utilization in Semi-Supervised Learning](https://mlsanity.com/paper/2412.10741/) (2024)

## Abstract (arXiv)

We address the challenge of training Vision Transformers (ViTs) when labeled data is scarce but unlabeled data is abundant. We propose Semi-Supervised Masked Autoencoder (SSMAE), a framework that jointly optimizes masked image reconstruction and classification using both unlabeled and labeled samples with dynamically selected pseudo-labels. SSMAE introduces a validation-driven gating mechanism that activates pseudo-labeling only after the model achieves reliable, high-confidence predictions that are consistent across both weakly and strongly augmented views of the same image, reducing confirmation bias. On CIFAR-10 and CIFAR-100, SSMAE consistently outperforms supervised ViT and fine-tuned MAE, with the largest gains in low-label regimes (+9.24% over ViT on CIFAR-10 with 10% labels). Our results demonstrate that when pseudo-labels are introduced is as important as how they are generated for data-efficient transformer training. Codes are available at https://github.com/atik666/ssmae.