mlsanityData to 14 Oct 2026

A Unified View of Masked Image Modeling

arXiv 2210.10615 ↗First posted 19 Oct 2022cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks. In this work, we propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce a simple yet effective method, termed as MaskDistill, which reconstructs normalized semantic features from teacher models at the masked positions, conditioning on corrupted input images. Experimental results on image classification and semantic segmentation show that MaskDistill achieves comparable or superior performance than state-of-the-art methods. When using the huge vision Transformer and pretraining 300 epochs, MaskDistill obtains 88.3% fine-tuning top-1 accuracy on ImageNet-1k (224 size) and 58.8% semantic segmentation mIoU metric on ADE20k (512 size). The code and pretrained models will be available at https://aka.ms/unimim.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 56 of its 60 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

56 of 60 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2019Natural Adversarial Examplescited by 1,408
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  13. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  14. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  15. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  17. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  18. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  19. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  20. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  21. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  22. 2021Intriguing Properties of Vision Transformerscited by 446
  23. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  24. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  25. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  26. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  27. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  28. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  29. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  30. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  31. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  32. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  33. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  34. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  35. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  36. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  37. 2022Bootstrapped Masked Autoencoders for Vision BERT Pretrainingcited by 60
  38. 2022SdAE: Self-distillated Masked Autoencodercited by 52

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 18; 2024: 10; 2025: 3; 2026: 3.

36 papers.

Sort:

By area:

  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  2. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  3. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  4. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  5. 2023PixMIM: Rethinking Pixel Reconstruction in Masked Image Modelingcited by 28
  6. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 27
  7. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  8. 2023Improving Pixel-based MIM by Reducing Wasted Modeling Capabilitycited by 24
  9. 2023Supervised Masked Knowledge Distillation for Few-Shot Transformerscited by 23
  10. 2025RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretationcited by 16
  11. 2024Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-trainingcited by 14
  12. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
24 more
  1. 2024Emerging Property of Masked Token for Effective Pre-trainingcited by 10
  2. 2024SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretrainingcited by 9
  3. 2023The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformerscited by 8
  4. 2023Delving Deeper into Data Scaling in Masked Image Modelingcited by 5
  5. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Modelscited by 5
  6. 2023MOMA:Distill from Self-Supervised Teacherscited by 4
  7. 2023RevColV2: Exploring Disentangled Representations in Masked Image Modelingcited by 4
  8. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  9. 2023Masking meets Supervision: A Strong Learning Alliancecited by 3
  10. 2023Remote Sensing Scene Classification with Masked Image Modeling (MIM)cited by 2
  11. 2023Enhancing Representations through Heterogeneous Self-Supervised Learningcited by 2
  12. 2023Asymmetric Masked Distillation for Pre-Training Small Foundation Modelscited by 2
  13. 2024SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representationcited by 2
  14. 2024Self-Supervised Learning for Real-World Object Detection: a Surveycited by 2
  15. 2023CeBed: A Benchmark for Deep Data-Driven OFDM Channel Estimationcited by 0
  16. 2023E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizercited by 0
  17. 2024Motion Guided Token Compression for Efficient Masked Video Modelingcited by 0
  18. 2024VIPriors 4: Visual Inductive Priors for Data-Efficient Deep Learning Challengescited by 0
  19. 2024Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integrationcited by 0
  20. 2024Stochastic Layer-Wise Shuffle for Improving Vision Mamba Trainingcited by 0
  21. 2025Data-Efficient Challenges in Visual Inductive Priors: A Retrospectivecited by 0
  22. 2026MEDiC: Multi-objective Exploration of Distillation from CLIPcited by 0
  23. 2026ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modelingcited by 0
  24. 2026GTR: Gated Token Recurrence for Efficient Dense Predictioncited by 0