mlsanityData to 14 Oct 2026

What to Hide from Your Students: Attention-Guided Masked Image Modeling

arXiv 2203.12719 ↗First posted 23 Mar 2022cs.CVcited by 118

Not on the map. Why?

Abstract (arXiv)

Transformers and masked language modeling are quickly being adopted and explored in computer vision as vision transformers and masked image modeling (MIM). In this work, we argue that image token masking differs from token masking in text, due to the amount and correlation of tokens in an image. In particular, to generate a challenging pretext task for MIM, we advocate a shift from random masking to informed masking. We develop and exhibit this idea in the context of distillation-based MIM, where a teacher transformer encoder generates an attention map, which we use to guide masking for the student. We thus introduce a novel masking strategy, called attention-guided masking (AttMask), and we demonstrate its effectiveness over random masking for dense distillation-based MIM as well as plain distillation-based self-supervised learning on classification tokens. We confirm that AttMask accelerates the learning process and improves the performance on a variety of downstream tasks. We provide the implementation code at https://github.com/gkakogeorgiou/attmask.

Key facts. Cited by 118 papers in the map (data through 14 Oct 2026). Builds on 63 of its 78 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

63 of 78 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2017Mask R-CNNcited by 7,277
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2014Going Deeper with Convolutionscited by 4,241
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  9. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  13. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  14. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  15. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  16. 2020Designing Network Design Spacescited by 960
  17. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  18. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  19. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  20. 2018Image Transformercited by 772
  21. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  22. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  23. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  24. 2017Look, Listen and Learncited by 458
  25. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  26. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  27. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  28. 2019Axial Attention in Multidimensional Transformerscited by 358
  29. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  30. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  31. 2020A Metric Learning Reality Checkcited by 321
  32. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  33. 2018DropBlock: A regularization method for convolutional networkscited by 262
  34. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  35. 2014Efficient Object Localization Using Convolutional Networkscited by 227
  36. 2017Unsupervised Representation Learning by Sorting Sequencescited by 221
  37. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  38. 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 191

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 33; 2024: 30; 2025: 23; 2026: 16.

118 papers.

Sort:

By area:

  1. 2022MIC: Masked Image Consistency for Context-Enhanced Domain Adaptationcited by 178
  2. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 166
  3. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  4. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  5. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  6. 2023Hard Patches Mining for Masked Image Modelingcited by 57
  7. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  8. 2024DINO-Foresight: Looking into the Future with DINOcited by 45
  9. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 45
  10. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 43
  11. 2023Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled videocited by 40
  12. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
38 more
  1. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  2. 2023HAP: Structure-Aware Masked Image Modeling for Human-Centric Perceptioncited by 33
  3. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  4. 2024Masked Image Modeling: A Surveycited by 29
  5. 2023Anatomical Invariance Modeling and Semantic Alignment for Self-supervised Learning in 3D Medical Image Analysiscited by 26
  6. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  7. 2023MAPSeg: Unified Unsupervised Domain Adaptation for Heterogeneous Medical Image Segmentation Based on 3D Masked Autoencoding and Pseudo-Labelingcited by 25
  8. 2022Attentive Mask CLIPcited by 23
  9. 2024FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learningcited by 23
  10. 2023PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videoscited by 22
  11. 2024MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embeddingcited by 22
  12. 2025Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learningcited by 21
  13. 2023Improving Masked Autoencoders by Learning Where to Maskcited by 20
  14. 2023COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformerscited by 17
  15. 2025EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generationcited by 17
  16. 2023Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?cited by 16
  17. 2022EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokenscited by 15
  18. 2024DetailCLIP: Detail-Oriented CLIP for Fine-Grained Taskscited by 15
  19. 2023GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networkscited by 14
  20. 2024Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-trainingcited by 14
  21. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  22. 2024Efficient Vision-Language Pre-training by Cluster Maskingcited by 13
  23. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 13
  24. 2022SERE: Exploring Feature Self-relation for Self-supervised Transformercited by 12
  25. 2022Exploring the Coordination of Frequency and Attention in Masked Image Modelingcited by 12
  26. 2023Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimickingcited by 12
  27. 2023Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Trainingcited by 12
  28. 2024Harnessing Massive Satellite Imagery with Efficient Masked Image Modelingcited by 12
  29. 2024A foundation model for generalizable disease diagnosis in chest X-ray imagescited by 12
  30. 2024InfMAE: A Foundation Model in the Infrared Modalitycited by 11
  31. 2025Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiencycited by 11
  32. 2024Emerging Property of Masked Token for Effective Pre-trainingcited by 10
  33. 2024AnatoMask: Enhancing Medical Image Segmentation with Reconstruction-guided Self-maskingcited by 10
  34. 2023MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignmentscited by 9
  35. 2024SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretrainingcited by 9
  36. 2024ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoderscited by 8
  37. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 8
  38. 2025M-SpecGene: Generalized Foundation Model for RGBT Multispectral Visioncited by 8

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Morphing Tokens Draw Strong Masked Image Models
  2. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models
  3. 2023The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers
  4. 2021Data Efficient Masked Language Modeling for Vision and Language
  5. 2025Rethinking Random Masking in Self-Distillation on ViT
  6. 2022Uniform Masking Prevails in Vision-Language Pretraining
  7. 2022Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoders
  8. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers