# CL-MAE: Curriculum-Learned Masked Autoencoders

arXiv: https://arxiv.org/abs/2308.16572 · HTML: https://mlsanity.com/paper/2308.16572/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 61 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers](https://mlsanity.com/paper/2111.12710/) (2021)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Unsupervised Learning of Dense Visual Representations](https://mlsanity.com/paper/2011.05499/) (2020)

## Built on by (most cited first, 22 of 22)

- [Curriculum Direct Preference Optimization for Diffusion and Consistency Models](https://mlsanity.com/paper/2405.13637/) (2024)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers](https://mlsanity.com/paper/2404.09326/) (2024)
- [CBM: Curriculum by Masking](https://mlsanity.com/paper/2407.05193/) (2024)
- [Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text](https://mlsanity.com/paper/2502.12953/) (2025)
- [Self-Guided Masked Autoencoder](https://mlsanity.com/paper/2507.19773/) (2025)
- [Understanding and Enhancing Mask-Based Pretraining towards Universal Representations](https://mlsanity.com/paper/2509.21650/) (2025)
- [Unlocking Efficiency: Adaptive Masking for Gene Transformer Models](https://mlsanity.com/paper/2408.07180/) (2024)
- [From Pixels to Components: Eigenvector Masking for Visual Representation Learning](https://mlsanity.com/paper/2502.06314/) (2025)
- [CuMoLoS-MAE: A Masked Autoencoder for Remote Sensing Data Reconstruction](https://mlsanity.com/paper/2508.14957/) (2025)
- [Structured-Noise Masked Modeling for Video, Audio and Beyond](https://mlsanity.com/paper/2503.16311/) (2025)
- [CUFG: Curriculum Unlearning Guided by the Forgetting Gradient](https://mlsanity.com/paper/2509.14633/) (2025)
- [Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization](https://mlsanity.com/paper/2402.18128/) (2024)
- [Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection](https://mlsanity.com/paper/2505.08561/) (2025)
- [Curriculum Multi-Task Self-Supervision Improves Lightweight Architectures for Onboard Satellite Hyperspectral Image Segmentation](https://mlsanity.com/paper/2509.13229/) (2025)
- [M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer](https://mlsanity.com/paper/2509.18005/) (2025)
- [CoMA: Complementary Masking and Hierarchical Dynamic Multi-Window Self-Attention in a Unified Pre-training Framework](https://mlsanity.com/paper/2511.05929/) (2025)
- [Collapse of Patches: Ranking Image Patches for Efficient Visual Modeling](https://mlsanity.com/paper/2511.22281/) (2025)
- [Curriculum-DPO++: Direct Preference Optimization via Data and Model Curricula for Text-to-Image Generation](https://mlsanity.com/paper/2602.13055/) (2026)
- [SPOTR: Spatio-temporal Pooling One-Token Reconstruction for Universal Physiological Signal Self-supervised Learning](https://mlsanity.com/paper/2606.21973/) (2026)
- [LocalProp: Neuro-Localized Memory-Efficient Backpropagation](https://mlsanity.com/paper/2609.32517/) (2026)
- [NV-Tesseract-AD: Curriculum-Enhanced Conditional Diffusion for Multivariate Time-Series Anomaly Detection](https://mlsanity.com/paper/2610.12754/) (2026)

## Abstract (arXiv)

Masked image modeling has been demonstrated as a powerful pretext task for generating robust representations that can be effectively generalized across multiple downstream tasks. Typically, this approach involves randomly masking patches (tokens) in input images, with the masking strategy remaining unchanged during training. In this paper, we propose a curriculum learning approach that updates the masking strategy to continually increase the complexity of the self-supervised reconstruction task. We conjecture that, by gradually increasing the task complexity, the model can learn more sophisticated and transferable representations. To facilitate this, we introduce a novel learnable masking module that possesses the capability to generate masks of different complexities, and integrate the proposed module into masked autoencoders (MAE). Our module is jointly trained with the MAE, while adjusting its behavior during training, transitioning from a partner to the MAE (optimizing the same reconstruction loss) to an adversary (optimizing the opposite loss), while passing through a neutral state. The transition between these behaviors is smooth, being regulated by a factor that is multiplied with the reconstruction loss of the masking module. The resulting training procedure generates an easy-to-hard curriculum. We train our Curriculum-Learned Masked Autoencoder (CL-MAE) on ImageNet and show that it exhibits superior representation learning capabilities compared to MAE. The empirical results on five downstream tasks confirm our conjecture, demonstrating that curriculum learning can be successfully used to self-supervise masked autoencoders. We release our code at https://github.com/ristea/cl-mae.