mlsanityData to 14 Oct 2026

Self-Guided Masked Autoencoder

arXiv 2507.19773 ↗First posted 26 Jul 2025cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Masked Autoencoder (MAE) is a self-supervised approach for representation learning, widely applicable to a variety of downstream tasks in computer vision. In spite of its success, it is still not fully uncovered what and how MAE exactly learns. In this paper, with an in-depth analysis, we discover that MAE intrinsically learns pattern-based patch-level clustering from surprisingly early stages of pretraining. Upon this understanding, we propose self-guided masked autoencoder, which internally generates informed mask by utilizing its progress in patch clustering, substituting the naive random masking of the vanilla MAE. Our approach significantly boosts its learning process without relying on any external models or supplementary information, keeping the benefit of self-supervised nature of MAE intact. Comprehensive experiments on various downstream tasks verify the effectiveness of the proposed method.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 51 of its 62 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

51 of 62 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2017Mask R-CNNcited by 7,277
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  11. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
38 more
  1. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  2. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  3. 2020Quantifying Attention Flow in Transformerscited by 775
  4. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  5. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  6. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  7. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  8. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  9. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  10. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  11. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  12. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 167
  13. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  14. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  15. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  16. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  17. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  18. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  19. 2021SiT: Self-supervised vIsion Transformercited by 99
  20. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  21. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  22. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  23. 2023Siamese Masked Autoencoderscited by 76
  24. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 71
  25. 2023Hard Patches Mining for Masked Image Modelingcited by 57
  26. 2023Masked Image Modeling with Local Multi-Scale Reconstructioncited by 43
  27. 2023MGMAE: Motion Guided Masking for Video Masked Autoencodingcited by 41
  28. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  29. 2022How to Understand Masked Autoencoderscited by 38
  30. 2022Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detectioncited by 33
  31. 2022mc-BEiT: Multi-choice Discretization for Image BERT Pre-trainingcited by 31
  32. 2022Less is More: Consistent Video Depth Estimation with Masked Frames Modelingcited by 30
  33. 2023PixMIM: Rethinking Pixel Reconstruction in Masked Image Modelingcited by 28
  34. 2023CL-MAE: Curriculum-Learned Masked Autoencoderscited by 22
  35. 2022SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learnerscited by 21
  36. 2023Improving Masked Autoencoders by Learning Where to Maskcited by 20
  37. 2022Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscited by 18
  38. 2022Boundary-aware Self-supervised Learning for Video Scene Segmentationcited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 13
  2. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  3. 2026Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learningcited by 0