mlsanityData to 14 Oct 2026

From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations

arXiv 2508.15404 ↗First posted 21 Aug 2025cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Masked Autoencoders (MAEs) have emerged as a powerful pretraining technique for vision foundation models. Despite their effectiveness, they require extensive hyperparameter tuning (masking ratio, patch size, encoder/decoder layers) when applied to novel datasets. While prior theoretical works have analyzed MAEs in terms of their attention patterns and hierarchical latent variable models, the connection between MAE hyperparameters and performance on downstream tasks is relatively unexplored. This work investigates how MAEs learn spatial correlations in the input image. We analytically derive the features learned by a linear MAE and show that masking ratio and patch size can be used to select for features that capture short- and long-range spatial correlations. We extend this analysis to non-linear MAEs to show that MAE representations adapt to spatial correlations in the dataset, beyond second-order statistics. Finally, we discuss some insights on how to select MAE hyper-parameters in practice.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

40 of 57 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023Segment Anythingcited by 8,234
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  11. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
28 more
  1. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  2. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  3. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  4. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  5. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  7. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  8. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  9. 2023Generalization in diffusion models arises from geometry-adaptive harmonic representationscited by 185
  10. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  11. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  12. 2023FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimationcited by 114
  13. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  14. 2019Robust and interpretable blind image denoising via bias-free convolutional neural networkscited by 90
  15. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  16. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 71
  17. 2024Learning by Reconstruction Produces Uninformative Features For Perceptioncited by 55
  18. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  19. 2022How to Understand Masked Autoencoderscited by 38
  20. 2022Masked Autoencoders Enable Efficient Knowledge Distillerscited by 36
  21. 2024Denoising: A Powerful Building-Block for Imaging, Inverse Problems, and Machine Learningcited by 35
  22. 2022The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Trainingcited by 33
  23. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 31
  24. 2023Motion-Guided Masking for Spatiotemporal Representation Learningcited by 30
  25. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  26. 2024Masked Image Modeling: A Surveycited by 29
  27. 2024Many Perception Tasks are Highly Redundant Functions of their Input Datacited by 4
  28. 2024Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning Representations from Incomplete EHR Data with Dual-Masked Autoencodingcited by 0
  2. 2026Diagonal Attenuation: A Finite-Sample Correction for PCAcited by 0