mlsanityData to 7 Oct 2026

Masked Siamese ConvNets

arXiv 2206.07700 ↗First posted 15 Jun 2022cs.CV cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning has shown superior performances over supervised methods on various vision benchmarks. The siamese network, which encourages embeddings to be invariant to distortions, is one of the most successful self-supervised visual representation learning approaches. Among all the augmentation methods, masking is the most general and straightforward method that has the potential to be applied to all kinds of input and requires the least amount of domain knowledge. However, masked siamese networks require particular inductive bias and practically only work well with Vision Transformers. This work empirically studies the problems behind masked siamese networks with ConvNets. We propose several empirical designs to overcome these problems gradually. Our method performs competitively on low-shot image classification and outperforms previous methods on object detection benchmarks. We discuss several remaining issues and hope this work can provide useful data points for future general-purpose self-supervised learning.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

41 of 54 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
29 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  13. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  14. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  16. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  17. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  18. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  19. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  20. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  21. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  22. 2017Large Batch Training of Convolutional Networkscited by 529
  23. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  24. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  25. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  26. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  27. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  28. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  29. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 38

Built on by

Cited by year

Citing papers per year: 2022: 9; 2023: 9; 2024: 3; 2025: 1.

22 papers.

Sort:

  1. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  2. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  3. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  4. 2023CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understandingcited by 48
  5. 2022Attentive Mask CLIPcited by 23
  6. 2022SimVTP: Simple Video Text Pre-training with Masked Autoencoderscited by 21
  7. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  8. 2022MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistencycited by 12
  9. 2022Minimalistic Unsupervised Learning with the Sparse Manifold Transformcited by 6
  10. 2022Self-Supervised Learning based on Heat Equationcited by 4
  11. 2023Modularity in Deep Learning: A Surveycited by 3
  12. 2023Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariancecited by 2
10 more
  1. 2023Pathway toward prior knowledge-integrated machine learning in engineeringcited by 2
  2. 2022MixMask: Revisiting Masking Strategy for Siamese ConvNetscited by 1
  3. 2023Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Wherecited by 1
  4. 2023Exploring Invariance in Images through One-way Wave Equationscited by 1
  5. 2022IDEA: Interactive DoublE Attentions from Label Embedding for Text Classificationcited by 0
  6. 2023Pair-wise Layer Attention with Spatial Masking for Video Predictioncited by 0
  7. 2024Machine Learning in Proton Exchange Membrane Water Electrolysis -- Part I: A Knowledge-Integrated Frameworkcited by 0
  8. 2024Masked Image Modeling as a Framework for Self-Supervised Learning across Eye Movementscited by 0
  9. 2024Relational Contrastive Learning and Masked Image Modeling for Scene Text Recognitioncited by 0
  10. 2025Scaling Non-Parametric Sampling with Representationcited by 0