mlsanityData to 7 Oct 2026

SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learners

arXiv 2205.14540 ↗First posted 28 May 2022cs.CV cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Recently, self-supervised Masked Autoencoders (MAE) have attracted unprecedented attention for their impressive representation learning ability. However, the pretext task, Masked Image Modeling (MIM), reconstructs the missing local patches, lacking the global understanding of the image. This paper extends MAE to a fully supervised setting by adding a supervised classification branch, thereby enabling MAE to learn global features from golden labels effectively. The proposed Supervised MAE (SupMAE) only exploits a visible subset of image patches for classification, unlike the standard supervised pre-training where all image patches are used. Through experiments, we demonstrate that SupMAE is not only more training efficient but it also learns more robust and transferable features. Specifically, SupMAE achieves comparable performance with MAE using only 30% of compute when evaluated on ImageNet with the ViT-B/16 model. SupMAE's robustness on ImageNet variants and transfer learning performance outperforms MAE and standard supervised pre-training counterparts. Codes are available at https://github.com/enyac-group/supmae.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

45 of 67 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
33 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2020Supervised Contrastive Learningcited by 2,803
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  17. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  18. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  19. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  20. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  21. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  22. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  23. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  24. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  25. 2022DeiT III: Revenge of the ViTcited by 417
  26. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  27. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  28. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  29. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  30. 2021Self-Supervised Learning with Swin Transformerscited by 121
  31. 2022Architecture-Agnostic Masked Image Modeling -- From ViT back to CNNcited by 31
  32. 2022RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-trainingcited by 24
  33. 2016Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classificationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 2; 2025: 7; 2026: 3.

21 papers.

Sort:

By area:

  1. 2022Language Modelling with Pixelscited by 53
  2. 2022Multitask Vision-Language Prompt Tuningcited by 51
  3. 2025Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learningcited by 33
  4. 2022Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Informationcited by 31
  5. 2024Masked Image Modeling: A Surveycited by 28
  6. 2023Masked Autoencoders are Efficient Class Incremental Learnerscited by 10
  7. 2023Masked Autoencoding Does Not Help Natural Language Supervision at Scalecited by 7
  8. 2025Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Modelscited by 6
  9. 2025ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learningcited by 6
  10. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  11. 2023Masking meets Supervision: A Strong Learning Alliancecited by 3
  12. 2025What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalizationcited by 3
9 more
  1. 2025Self-Guided Masked Autoencodercited by 3
  2. 2023Revisiting Hidden Representations in Transfer Learning for Medical Imagingcited by 2
  3. 2023Remote Sensing Scene Classification with Masked Image Modeling (MIM)cited by 2
  4. 2024Hybrid diffusion models: combining supervised and generative pretraining for label-efficient fine-tuning of segmentation modelscited by 2
  5. 2025Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognitioncited by 0
  6. 2025Assessing Foundation Models for Mold Colony Detection with Limited Training Datacited by 0
  7. 2026Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Datacited by 0
  8. 2026Expectation and Acoustic Neural Network Representations Enhance Music Identification from Brain Activitycited by 0
  9. 2026MiTHras: Task-specific Hierarchical Semi-supervised Contrastive Masked Autoencoder for Mitotic Figure Analysiscited by 0