# Stare at What You See: Masked Image Modeling without Reconstruction

arXiv: https://arxiv.org/abs/2211.08887 · HTML: https://mlsanity.com/paper/2211.08887/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 58 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://mlsanity.com/paper/2208.06366/) (2022)

## Built on by (most cited first, 30 of 30)

- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning](https://mlsanity.com/paper/2403.09634/) (2024)
- [Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation](https://mlsanity.com/paper/2306.04811/) (2023)
- [Rethinking Patch Dependence for Masked Autoencoders](https://mlsanity.com/paper/2401.14391/) (2024)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modeling](https://mlsanity.com/paper/2301.01006/) (2023)
- [IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training](https://mlsanity.com/paper/2310.07355/) (2023)
- [Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition](https://mlsanity.com/paper/2310.04999/) (2023)
- [Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training](https://mlsanity.com/paper/2404.08327/) (2024)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment](https://mlsanity.com/paper/2604.12012/) (2026)
- [Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training](https://mlsanity.com/paper/2312.02914/) (2023)
- [Beyond Image Borders: Learning Feature Extrapolation for Unbounded Image Composition](https://mlsanity.com/paper/2309.12042/) (2023)
- [Emerging Property of Masked Token for Effective Pre-training](https://mlsanity.com/paper/2404.08330/) (2024)
- [MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments](https://mlsanity.com/paper/2307.09361/) (2023)
- [PELA: Learning Parameter-Efficient Models with Low-Rank Approximation](https://mlsanity.com/paper/2310.10700/) (2023)
- [Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners](https://mlsanity.com/paper/2306.15876/) (2023)
- [Mask to reconstruct: Cooperative Semantics Completion for Video-text Retrieval](https://mlsanity.com/paper/2305.07910/) (2023)
- [Self-Supervised Learning with Generative Adversarial Networks for Electron Microscopy](https://mlsanity.com/paper/2402.18286/) (2024)
- [Aerial Image Object Detection With Vision Transformer Detector (ViTDet)](https://mlsanity.com/paper/2301.12058/) (2023)
- [Remote Sensing Scene Classification with Masked Image Modeling (MIM)](https://mlsanity.com/paper/2302.14256/) (2023)
- [How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?](https://mlsanity.com/paper/2409.00543/) (2024)
- [Self-Supervised Learning for Real-World Object Detection: a Survey](https://mlsanity.com/paper/2410.07442/) (2024)
- [Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning](https://mlsanity.com/paper/2312.02194/) (2023)
- [Self-distilled Masked Attention guided masked image modeling with noise Regularized Teacher (SMART) for medical image analysis](https://mlsanity.com/paper/2310.01209/) (2023)
- [Enhancing Vision-Language Model with Unmasked Token Alignment](https://mlsanity.com/paper/2405.19009/) (2024)
- [C^2DA: Contrastive and Context-aware Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2410.19748/) (2024)
- [Collapse of Patches: Ranking Image Patches for Efficient Visual Modeling](https://mlsanity.com/paper/2511.22281/) (2025)
- [Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance](https://mlsanity.com/paper/2604.16086/) (2026)
- [Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks](https://mlsanity.com/paper/2605.18491/) (2026)

## Abstract (arXiv)

Masked Autoencoders (MAE) have been prevailing paradigms for large-scale vision representation pre-training. By reconstructing masked image patches from a small portion of visible image regions, MAE forces the model to infer semantic correlation within an image. Recently, some approaches apply semantic-rich teacher models to extract image features as the reconstruction target, leading to better performance. However, unlike the low-level features such as pixel values, we argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact image.This raises one question: is reconstruction necessary in Masked Image Modeling (MIM) with a teacher model? In this paper, we propose an efficient MIM paradigm named MaskAlign. MaskAlign simply learns the consistency of visible patch features extracted by the student model and intact image features extracted by the teacher model. To further advance the performance and tackle the problem of input inconsistency between the student and teacher model, we propose a Dynamic Alignment (DA) module to apply learnable alignment. Our experimental results demonstrate that masked modeling does not lose effectiveness even without reconstruction on masked regions. Combined with Dynamic Alignment, MaskAlign can achieve state-of-the-art performance with much higher efficiency. Code and models will be available at https://github.com/OpenPerceptionX/maskalign.