mlsanityData to 13 Oct 2026

Beyond [cls]: Exploring the true potential of Masked Image Modeling representations

arXiv 2412.03215 ↗First posted 4 Dec 2024cs.CV cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Masked Image Modeling (MIM) has emerged as a promising approach for Self-Supervised Learning (SSL) of visual representations. However, the out-of-the-box performance of MIMs is typically inferior to competing approaches. Most users cannot afford fine-tuning due to the need for large amounts of data, high GPU consumption, and specialized user knowledge. Therefore, the practical use of MIM representations is limited. In this paper we ask what is the reason for the poor out-of-the-box performance of MIMs. Is it due to weaker features produced by MIM models, or is it due to suboptimal usage? Through detailed analysis, we show that attention in MIMs is spread almost uniformly over many patches, leading to ineffective aggregation by the [cls] token. Based on this insight, we propose Selective Aggregation to better capture the rich semantic information retained in patch tokens, which significantly improves the out-of-the-box performance of MIM.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 59 of its 80 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

59 of 80 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,860
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,850
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,285
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,826
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
  10. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,590
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,381
  12. 2014How transferable are features in deep neural networks?cited by 1,219
  13. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,150
  14. 2016Colorful Image Colorizationcited by 1,114
  15. 2018Attention-based Deep Multiple Instance Learningcited by 1,083
  16. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,064
  17. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,018
  18. 2020What Makes for Good Views for Contrastive Learning?cited by 844
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 574
  20. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 514
  21. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  22. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  23. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  24. 2023A Cookbook of Self-Supervised Learningcited by 249
  25. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  26. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  27. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  28. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  29. 2020Evaluating Weakly Supervised Object Localization Methods Rightcited by 105
  30. 2021SiT: Self-supervised vIsion Transformercited by 99
  31. 2023What Do Self-Supervised Vision Transformers Learn?cited by 91
  32. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  33. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  34. 2024Learning by Reconstruction Produces Uninformative Features For Perceptioncited by 55
  35. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  36. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
  37. 2022Exploring Target Representations for Masked Autoencoderscited by 46
  38. 2023Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled videocited by 40

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiencycited by 11
  2. 2026Pretext Matters: An Empirical Study of SSL Methods in Medical Imagingcited by 2
  3. 2026SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Modelscited by 1
  4. 2025TICON: A Slide-Level Tile Contextualizer for Histopathology Representation Learningcited by 0
  5. 2026Channel-Aware Probing for Multi-Channel Imagingcited by 0
  6. 2026Suppressing Non-Semantic Noise in Masked Image Modeling Representationscited by 0