# A Unified View of Masked Image Modeling

arXiv: https://arxiv.org/abs/2210.10615 · HTML: https://mlsanity.com/paper/2210.10615/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 60 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)

## Built on by (most cited first, 30 of 36)

- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [Distilling Large Vision-Language Model with Out-of-Distribution Generalizability](https://mlsanity.com/paper/2307.03135/) (2023)
- [Stare at What You See: Masked Image Modeling without Reconstruction](https://mlsanity.com/paper/2211.08887/) (2022)
- [PixMIM: Rethinking Pixel Reconstruction in Masked Image Modeling](https://mlsanity.com/paper/2303.02416/) (2023)
- [UNIC: Universal Classification Models via Multi-teacher Distillation](https://mlsanity.com/paper/2408.05088/) (2024)
- [CAE v2: Context Autoencoder with CLIP Target](https://mlsanity.com/paper/2211.09799/) (2022)
- [Improving Pixel-based MIM by Reducing Wasted Modeling Capability](https://mlsanity.com/paper/2308.00261/) (2023)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretation](https://mlsanity.com/paper/2504.03166/) (2025)
- [Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training](https://mlsanity.com/paper/2404.08327/) (2024)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Emerging Property of Masked Token for Effective Pre-training](https://mlsanity.com/paper/2404.08330/) (2024)
- [SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining](https://mlsanity.com/paper/2404.01156/) (2024)
- [The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers](https://mlsanity.com/paper/2302.10494/) (2023)
- [Delving Deeper into Data Scaling in Masked Image Modeling](https://mlsanity.com/paper/2305.15248/) (2023)
- [ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models](https://mlsanity.com/paper/2504.00037/) (2025)
- [MOMA:Distill from Self-Supervised Teachers](https://mlsanity.com/paper/2302.02089/) (2023)
- [RevColV2: Exploring Disentangled Representations in Masked Image Modeling](https://mlsanity.com/paper/2309.01005/) (2023)
- [Aerial Image Object Detection With Vision Transformer Detector (ViTDet)](https://mlsanity.com/paper/2301.12058/) (2023)
- [Masking meets Supervision: A Strong Learning Alliance](https://mlsanity.com/paper/2306.11339/) (2023)
- [Remote Sensing Scene Classification with Masked Image Modeling (MIM)](https://mlsanity.com/paper/2302.14256/) (2023)
- [Enhancing Representations through Heterogeneous Self-Supervised Learning](https://mlsanity.com/paper/2310.05108/) (2023)
- [Asymmetric Masked Distillation for Pre-Training Small Foundation Models](https://mlsanity.com/paper/2311.03149/) (2023)
- [SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation](https://mlsanity.com/paper/2406.10673/) (2024)
- [Self-Supervised Learning for Real-World Object Detection: a Survey](https://mlsanity.com/paper/2410.07442/) (2024)
- [CeBed: A Benchmark for Deep Data-Driven OFDM Channel Estimation](https://mlsanity.com/paper/2306.13761/) (2023)
- [E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizer](https://mlsanity.com/paper/2311.17267/) (2023)
- [Motion Guided Token Compression for Efficient Masked Video Modeling](https://mlsanity.com/paper/2402.18577/) (2024)
- [VIPriors 4: Visual Inductive Priors for Data-Efficient Deep Learning Challenges](https://mlsanity.com/paper/2406.18176/) (2024)

## Abstract (arXiv)

Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks. In this work, we propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce a simple yet effective method, termed as MaskDistill, which reconstructs normalized semantic features from teacher models at the masked positions, conditioning on corrupted input images. Experimental results on image classification and semantic segmentation show that MaskDistill achieves comparable or superior performance than state-of-the-art methods. When using the huge vision Transformer and pretraining 300 epochs, MaskDistill obtains 88.3% fine-tuning top-1 accuracy on ImageNet-1k (224 size) and 58.8% semantic segmentation mIoU metric on ADE20k (512 size). The code and pretrained models will be available at https://aka.ms/unimim.