mlsanityData to 14 Oct 2026

Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers

arXiv 2203.14313 ↗First posted 27 Mar 2022cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

The past year has witnessed a rapid development of masked image modeling (MIM). MIM is mostly built upon the vision transformers, which suggests that self-supervised visual representations can be done by masking input image parts while requiring the target model to recover the missing contents. MIM has demonstrated promising results on downstream tasks, yet we are interested in whether there exist other effective ways to `learn by recovering missing contents'. In this paper, we investigate this topic by designing five other learning objectives that follow the same procedure as MIM but degrade the input image in different ways. With extensive experiments, we manage to summarize a few design principles for token-based pre-training of vision transformers. In particular, the best practice is obtained by keeping the original image style and enriching spatial masking with spatial misalignment -- this design achieves superior performance over MIM in a series of downstream recognition tasks without extra computational cost. The code is available at https://github.com/sunsmarterjie/beyond_masking.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 51 of its 57 references that are in the map. First posted on arXiv 27 Mar 2022.

Builds on

51 of 57 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2017Mask R-CNNcited by 7,277
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2014Going Deeper with Convolutionscited by 4,241
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  14. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  15. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  16. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  17. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  18. 2016Colorful Image Colorizationcited by 1,114
  19. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  20. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  21. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  22. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  23. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  24. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  25. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  27. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  28. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  29. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  30. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  31. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  32. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  33. 2016Learning Representations for Automatic Colorizationcited by 251
  34. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  35. 2021Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformercited by 115
  36. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  37. 2021MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokenscited by 46
  38. 2021Glance-and-Gaze Vision Transformercited by 42

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 1; 2025: 2; 2026: 1.

14 papers.

Sort:

By area:

  1. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  2. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  3. 2022Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migrationcited by 24
  4. 2022HiViT: Hierarchical Vision Transformer Meets Masked Image Modelingcited by 23
  5. 2024Masked Angle-Aware Autoencoder for Remote Sensing Imagescited by 22
  6. 2023Correlational Image Modeling for Self-Supervised Visual Pre-Trainingcited by 11
  7. 2023Pre-training with Random Orthogonal Projection Image Modelingcited by 11
  8. 2023Deblurring Masked Autoencoder is Better Recipe for Ultrasound Image Recognitioncited by 8
  9. 2023Beyond Pretrained Features: Noisy Image Modeling Provides Adversarial Defensecited by 7
  10. 2026Self-Supervised Visual On-Policy Distillationcited by 6
  11. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  12. 2025Understanding and Enhancing Mask-Based Pretraining towards Universal Representationscited by 3
2 more
  1. 2023Remote Sensing Scene Classification with Masked Image Modeling (MIM)cited by 2
  2. 2025Prompt-based Dynamic Token Pruning for Efficient Segmentation of Medical Imagescited by 0