# Masked Image Modeling: A Survey

arXiv: https://arxiv.org/abs/2408.06687 · HTML: https://mlsanity.com/paper/2408.06687/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 193 of its 255 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 30 of 193)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)

## Built on by (most cited first, 29 of 29)

- [ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy](https://mlsanity.com/paper/2411.02572/) (2024)
- [Alligat0R: Pre-Training Through Co-Visibility Segmentation for Relative Camera Pose Regression](https://mlsanity.com/paper/2503.07561/) (2025)
- [AgriFM: A Multi-source Temporal Remote Sensing Foundation Model for Agriculture Mapping](https://mlsanity.com/paper/2505.21357/) (2025)
- [LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models](https://mlsanity.com/paper/2502.02069/) (2025)
- [Agentic AI in Remote Sensing: Foundations, Taxonomy, and Emerging Systems](https://mlsanity.com/paper/2601.01891/) (2026)
- [MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencoders for Earth Observation Tasks](https://mlsanity.com/paper/2505.14951/) (2025)
- [CLICv2: Image Complexity Representation via Content Invariance Contrastive Learning](https://mlsanity.com/paper/2503.06641/) (2025)
- [Sapiens2](https://mlsanity.com/paper/2604.21681/) (2026)
- [AMNS: Attention-Weighted Selective Mask and Noise Label Suppression for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2409.06385/) (2024)
- [From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations](https://mlsanity.com/paper/2508.15404/) (2025)
- [US-JEPA: A Joint Embedding Predictive Architecture for Ultrasound](https://mlsanity.com/paper/2602.19322/) (2026)
- [Multimodal Datasets with Controllable Mutual Information](https://mlsanity.com/paper/2510.21686/) (2025)
- [Conjuring Positive Pairs for Efficient Unification of Representation Learning and Image Synthesis](https://mlsanity.com/paper/2503.15060/) (2025)
- [Adaptive Mask-guided K-space Diffusion for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2506.18270/) (2025)
- [Curriculum Multi-Task Self-Supervision Improves Lightweight Architectures for Onboard Satellite Hyperspectral Image Segmentation](https://mlsanity.com/paper/2509.13229/) (2025)
- [Where Do Flow Semantics Reside? A Protocol-Native Tabular Pretraining Paradigm for Encrypted Traffic Classification](https://mlsanity.com/paper/2603.10051/) (2026)
- [CoRe-ECG: Advancing Self-Supervised Representation Learning for 12-Lead ECG via Contrastive and Reconstructive Synergy](https://mlsanity.com/paper/2604.11359/) (2026)
- [MTL-MAD: Multi-Task Learners are Effective Medical Anomaly Detectors](https://mlsanity.com/paper/2605.05891/) (2026)
- [HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learning](https://mlsanity.com/paper/2605.31068/) (2026)
- [Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation](https://mlsanity.com/paper/2606.01031/) (2026)
- [Structure-Guided Mixed Masked Pretraining and Spatial Continuity Regularization for Printed Circuit Board Defect Detection](https://mlsanity.com/paper/2606.03508/) (2026)
- [Visual Quality Score Assessment of Large White Goods in Remanufacture with Multi-View Deformable-DETR](https://mlsanity.com/paper/2606.14556/) (2026)
- [MuSViT: A Foundation Vision Model for Sheet Music Representation](https://mlsanity.com/paper/2606.31811/) (2026)
- [The JEPA Predictor: A Transferable Operator for Occluded Feature Completion](https://mlsanity.com/paper/2607.16274/) (2026)
- [Spectral Aliasing Pretext: A novel task for Self-Supervised fault diagnosis in rotating machinery](https://mlsanity.com/paper/2608.05705/) (2026)
- [Large Models for Battery Prognostics and Health Management: A Review and Future Roadmap](https://mlsanity.com/paper/2608.26111/) (2026)
- [A generalizable structural brain MRI foundation model built through dual-priority federated pretraining](https://mlsanity.com/paper/2609.27611/) (2026)
- [LocalProp: Neuro-Localized Memory-Efficient Backpropagation](https://mlsanity.com/paper/2609.32517/) (2026)
- [Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning](https://mlsanity.com/paper/2610.13915/) (2026)

## Abstract (arXiv)

In this work, we survey recent studies on masked image modeling (MIM), an approach that emerged as a powerful self-supervised learning technique in computer vision. The MIM task involves masking some information, e.g. pixels, patches, or even latent representations, and training a model, usually an autoencoder, to predicting the missing information by using the context available in the visible part of the input. We identify and formalize two categories of approaches on how to implement MIM as a pretext task, one based on reconstruction and one based on contrastive learning. Then, we construct a taxonomy and review the most prominent papers in recent years. We complement the manually constructed taxonomy with a dendrogram obtained by applying a hierarchical clustering algorithm. We further identify relevant clusters via manually inspecting the resulting dendrogram. Our review also includes datasets that are commonly used in MIM research. We aggregate the performance results of various masked image modeling methods on the most popular datasets, to facilitate the comparison of competing methods. Finally, we identify research gaps and propose several interesting directions of future work. We supplement our survey with the following public repository containing organized references: https://github.com/vladhondru25/MIM-Survey.