mlsanityData to 7 Oct 2026

Suppressing Non-Semantic Noise in Masked Image Modeling Representations

arXiv 2604.00172 ↗First posted 31 Mar 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Masked Image Modeling (MIM) has become a ubiquitous self-supervised vision paradigm. In this work, we show that MIM objectives cause the learned representations to retain non-semantic information, which ultimately hurts performance during inference. We introduce a model-agnostic score for semantic invariance using Principal Component Analysis (PCA) on real and synthetic non-semantic images. Based on this score, we propose a simple method, Semantically Orthogonal Artifact Projection (SOAP), to directly suppress non-semantic information in patch representations, leading to consistent improvements in zero-shot performance across various MIM-based models. SOAP is a post-hoc suppression method, requires zero training, and can be attached to any model as a single linear head.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

34 of 52 references.

References · 34

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
22 more
  1. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  2. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  3. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  5. 2023Vision Transformers Need Registerscited by 844
  6. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  9. 2022DeiT III: Revenge of the ViTcited by 417
  10. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 166
  11. 2022TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cutcited by 111
  12. 2023Three Pillars improving Vision Foundation Model Distillation for Lidarcited by 43
  13. 2024Denoising Vision Transformerscited by 40
  14. 2022Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Viewscited by 37
  15. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  16. 2025Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learningcited by 21
  17. 2024MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representationscited by 19
  18. 2024SINDER: Repairing the Singular Defects of DINOv2cited by 18
  19. 2023Stochastic positional embeddings improve masked image modelingcited by 12
  20. 2023Analyzing Local Representations of Self-supervised Vision Transformerscited by 8
  21. 2023DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistencycited by 6
  22. 2024Beyond [cls]: Exploring the true potential of Masked Image Modeling representationscited by 6

Built on by

Not cited yet.