# Robust Representation Learning in Masked Autoencoders

arXiv: https://arxiv.org/abs/2602.03531 · HTML: https://mlsanity.com/paper/2602.03531/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders](https://mlsanity.com/paper/2206.10207/) (2022)
- [ConvMAE: Masked Convolution Meets Masked Autoencoders](https://mlsanity.com/paper/2205.03892/) (2022)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [Understanding Masked Image Modeling via Learning Occlusion Invariant Feature](https://mlsanity.com/paper/2208.04164/) (2022)
- [Understanding Masked Autoencoders via Hierarchical Latent Variable Models](https://mlsanity.com/paper/2306.04898/) (2023)
- [How to Understand Masked Autoencoders](https://mlsanity.com/paper/2202.03670/) (2022)
- [Rethinking Patch Dependence for Masked Autoencoders](https://mlsanity.com/paper/2401.14391/) (2024)

## Abstract (arXiv)

Masked Autoencoders (MAEs) achieve impressive performance in image classification tasks, yet the internal representations they learn remain less understood. This work started as an attempt to understand the strong downstream classification performance of MAE. In this process we discover that representations learned with the pretraining and fine-tuning, are quite robust -- demonstrating a good classification performance in the presence of degradations, such as blur and occlusions. Through layer-wise analysis of token embeddings, we show that pretrained MAE progressively constructs its latent space in a class-aware manner across network depth: embeddings from different classes lie in subspaces that become increasingly separable. We further observe that MAE exhibits early and persistent global attention across encoder layers, in contrast to standard Vision Transformers (ViTs). To quantify feature robustness, we introduce two sensitivity indicators: directional alignment between clean and perturbed embeddings, and head-wise retention of active features under degradations. These studies help establish the robust classification performance of MAEs.