mlsanityData to 7 Oct 2026

Masked Siamese Networks for Label-Efficient Learning

arXiv 2204.07141 ↗First posted 14 Apr 2022cs.LG cs.AI cs.CVcited by 301

Not on the map. Why?

Abstract (arXiv)

We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available.

Key facts. Cited by 301 papers in the map (data through 7 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

44 of 53 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
32 more
  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2016Colorful Image Colorizationcited by 1,113
  17. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  18. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  19. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  20. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  21. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  22. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  23. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  24. 2016Learning Representations for Automatic Colorizationcited by 251
  25. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  26. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  27. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  28. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  29. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  30. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 64
  31. 2021Barely-Supervised Learning: Semi-Supervised Learning with very few labeled imagescited by 17
  32. 2019Cyanure: An Open-Source Toolbox for Empirical Risk Minimization for Python, C++, and soon morecited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 49; 2023: 95; 2024: 77; 2025: 47; 2026: 33.

301 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  3. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  4. 2023A Cookbook of Self-Supervised Learningcited by 248
  5. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  6. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219
  7. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 165
  8. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  9. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  10. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  11. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  12. 2022Parametric Classification for Generalized Category Discovery: A Baseline Studycited by 113
38 more
  1. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  2. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  3. 2023Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similaritiescited by 78
  4. 2023Siamese Masked Autoencoderscited by 75
  5. 2023Hyperbolic Deep Learning in Computer Vision: A Surveycited by 72
  6. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  7. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 70
  8. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  9. 2023No Representation Rules Them All in Category Discoverycited by 61
  10. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  11. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  12. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  13. 2023Diffusion Models Beat GANs on Image Classificationcited by 57
  14. 2024Scaling Wearable Foundation Modelscited by 57
  15. 2023Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learningcited by 55
  16. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  17. 2022Understanding Masked Image Modeling via Learning Occlusion Invariant Featurecited by 50
  18. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  19. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 48
  20. 2023RxRx1: A Dataset for Evaluating Experimental Batch Correction Methodscited by 45
  21. 2025SensorLM: Learning the Language of Wearable Sensorscited by 44
  22. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  23. 2024SpectralEarth: Training Hyperspectral Foundation Models at Scalecited by 42
  24. 2025Galileo: Learning Global & Local Features of Many Remote Sensing Modalitiescited by 41
  25. 2023Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labelscited by 37
  26. 2022Self-Guided Diffusion Modelscited by 36
  27. 2022A Unified View of Masked Image Modelingcited by 36
  28. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  29. 2023Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representationscited by 35
  30. 2023Exploring the Limits of Deep Image Clustering using Pretrained Modelscited by 33
  31. 2023Do text-free diffusion models learn discriminative visual representations?cited by 33
  32. 2022Architecture-Agnostic Masked Image Modeling -- From ViT back to CNNcited by 31
  33. 2022Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Informationcited by 31
  34. 2023Vision-Based Traffic Accident Detection and Anticipation: A Surveycited by 31
  35. 2024Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized Visual Class Discoverycited by 30
  36. 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 29
  37. 2024Masked Image Modeling: A Surveycited by 28
  38. 2024Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretrainingcited by 27

50 most cited of 301 All 301 (JSON)