mlsanityData to 14 Oct 2026

Object-wise Masked Autoencoders for Fast Pre-training

arXiv 2205.14338 ↗First posted 28 May 2022cs.CV cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Self-supervised pre-training for images without labels has recently achieved promising performance in image classification. The success of transformer-based methods, ViT and MAE, draws the community's attention to the design of backbone architecture and self-supervised task. In this work, we show that current masked image encoding models learn the underlying relationship between all objects in the whole scene, instead of a single object representation. Therefore, those methods bring a lot of compute time for self-supervised pre-training. To solve this issue, we introduce a novel object selection and division strategy to drop non-object patches for learning object-wise representations by selective reconstruction with interested region masks. We refer to this method ObjMAE. Extensive experiments on four commonly-used datasets demonstrate the effectiveness of our model in reducing the compute cost by 72% while achieving competitive performance. Furthermore, we investigate the inter-object and intra-object relationship and find that the latter is crucial for self-supervised pre-training.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

31 of 36 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
19 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  9. 2020Quantifying Attention Flow in Transformerscited by 775
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  12. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  13. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  14. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  15. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  16. 2021Self-Supervised Learning with Swin Transformerscited by 122
  17. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  18. 2021GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinementcited by 109
  19. 2015Efficient inference in occlusion-aware generative models of imagescited by 11

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 6; 2024: 5; 2025: 1.

15 papers.

Sort:

  1. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  2. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  3. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  4. 2024Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learningcited by 20
  5. 2024Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-trainingcited by 14
  6. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  7. 2024Emerging Property of Masked Token for Effective Pre-trainingcited by 10
  8. 2022Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methodscited by 8
  9. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  10. 2025M-SpecGene: Generalized Foundation Model for RGBT Multispectral Visioncited by 8
  11. 2023Masked Momentum Contrastive Learning for Zero-shot Semantic Understandingcited by 4
  12. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
3 more
  1. 2023Beyond Accuracy: Statistical Measures and Benchmark for Evaluation of Representation from Self-Supervised Learningcited by 3
  2. 2024DailyMAE: Towards Pretraining Masked Autoencoders in One Daycited by 2
  3. 2024The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learningcited by 1