mlsanityData to 13 Oct 2026

How to Understand Masked Autoencoders

arXiv 2202.03670 ↗First posted 8 Feb 2022cs.CV cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

"Masked Autoencoders (MAE) Are Scalable Vision Learners" revolutionizes the self-supervised learning method in that it not only achieves the state-of-the-art for image pre-training, but is also a milestone that bridges the gap between visual and linguistic masked autoencoding (BERT-style) pre-trainings. However, to our knowledge, to date there are no theoretical perspectives to explain the powerful expressivity of MAE. In this paper, we, for the first time, propose a unified theoretical framework that provides a mathematical understanding for MAE. Specifically, we explain the patch-based attention approaches of MAE using an integral kernel under a non-overlapping domain decomposition setting. To help the research community to further comprehend the main reasons of the great success of MAE, based on our framework, we pose five questions and answer them with mathematical rigor using insights from operator theory.

Key facts. Cited by 38 papers in the map (data through 13 Oct 2026). Builds on 28 of its 55 references that are in the map. First posted on arXiv 8 Feb 2022.

Builds on

28 of 55 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  8. 2016Layer Normalizationcited by 5,624
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  10. 2014Going Deeper with Convolutionscited by 4,239
  11. 2018Neural Ordinary Differential Equationscited by 3,750
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
16 more
  1. 2017Non-local Neural Networkscited by 2,291
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
  3. 2020Rethinking Attention with Performerscited by 1,594
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  5. 2020On Layer Normalization in the Transformer Architecturecited by 910
  6. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 674
  7. 2021Transformer in Transformercited by 633
  8. 2021Choose a Transformer: Fourier or Galerkincited by 340
  9. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  10. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  11. 2020Query-Key Normalization for Transformerscited by 270
  12. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  13. 2021Vision Transformers are Robust Learnerscited by 207
  14. 2020Toward Transformer-Based Object Detectioncited by 84
  15. 2020DeepGreen: Deep Learning of Green's Functions for Nonlinear Boundary Value Problemscited by 61
  16. 2019Multi-Graph Transformer for Free-Hand Sketch Recognitioncited by 35

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 8; 2023: 14; 2024: 6; 2025: 7; 2026: 3.

38 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
  3. 2023Scalable Transformer for PDE Surrogate Modelingcited by 125
  4. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  5. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  6. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  7. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  8. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  9. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  10. 2022Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscited by 18
  11. 2023Information Flow in Self-Supervised Learningcited by 16
  12. 2024Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-trainingcited by 14
26 more
  1. 2024Masked Autoencoders are PDE Learnerscited by 12
  2. 2022Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domainscited by 10
  3. 2023Image Deblurring by Exploring In-depth Properties of Transformercited by 10
  4. 2023Understanding Masked Autoencoders From a Local Contrastive Perspectivecited by 10
  5. 2024Emerging Property of Masked Token for Effective Pre-trainingcited by 10
  6. 2023BiLMa: Bidirectional Local-Matching for Text-based Person Re-identificationcited by 8
  7. 2023Understanding Self-Supervised Pretraining with Part-Aware Representation Learningcited by 7
  8. 2023Continuous Spatiotemporal Transformerscited by 7
  9. 2024On the Role of Discrete Tokenization in Visual Representation Learningcited by 7
  10. 2024How Molecules Impact Cells: Unlocking Contrastive PhenoMolecular Retrievalcited by 6
  11. 2025An Augmentation Overlap Theory of Contrastive Learningcited by 5
  12. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  13. 2025Self-Guided Masked Autoencodercited by 3
  14. 2023Hint-Aug: Drawing Hints from Foundation Vision Transformers Towards Boosted Few-Shot Parameter-Efficient Tuningcited by 2
  15. 2025From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlationscited by 2
  16. 2025MINR: Implicit Neural Representations with Masked Image Modellingcited by 1
  17. 2025Understanding and Enhancing Mask-Based Pretraining towards Universal Representationscited by 1
  18. 2023Hiding Data Helps: On the Benefits of Masking for Sparse Codingcited by 0
  19. 2023CSSL-MHTR: Continual Self-Supervised Learning for Scalable Multi-script Handwritten Text Recognitioncited by 0
  20. 2023Deep Tensor Networkcited by 0
  21. 2024KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encodercited by 0
  22. 2025MoCA: Multi-modal Cross-masked Autoencoder for Time Series in Digital Healthcited by 0
  23. 2025Foundation Artificial Intelligence Models for Health Recognition Using Face Photographs (FAHR-Face)cited by 0
  24. 2026Robust Representation Learning in Masked Autoencoderscited by 0
  25. 2026Self-explainable Operator Learning for Discovering Spatial Patterns in Functional Datacited by 0
  26. 2026A Theoretical Framework for Masked Pretraining (MPT)cited by 0