mlsanityData to 7 Oct 2026

Masked Contrastive Representation Learning

arXiv 2211.06012 ↗First posted 11 Nov 2022cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Masked image modelling (e.g., Masked AutoEncoder) and contrastive learning (e.g., Momentum Contrast) have shown impressive performance on unsupervised visual representation learning. This work presents Masked Contrastive Representation Learning (MACRL) for self-supervised visual pre-training. In particular, MACRL leverages the effectiveness of both masked image modelling and contrastive learning. We adopt an asymmetric setting for the siamese network (i.e., encoder-decoder structure in both branches), where one branch with higher mask ratio and stronger data augmentation, while the other adopts weaker data corruptions. We optimize a contrastive learning objective based on the learned features from the encoder in both branches. Furthermore, we minimize the L1 reconstruction loss according to the decoders' outputs. In our experiments, MACRL presents superior results on various vision benchmarks, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and two other ImageNet subsets. Our framework provides unified insights on self-supervised visual pre-training and future research.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 41 of its 48 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

41 of 48 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
29 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  13. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  14. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  17. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  18. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  19. 2017Large Batch Training of Convolutional Networkscited by 529
  20. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  21. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  22. 2020fastai: A Layered API for Deep Learningcited by 183
  23. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  24. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  25. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  26. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  27. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  28. 2022Exploring Target Representations for Masked Autoencoderscited by 46
  29. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 38

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1.

5 papers.

  1. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  2. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  3. 2023MOMA:Distill from Self-Supervised Teacherscited by 4
  4. 2023Masked Momentum Contrastive Learning for Zero-shot Semantic Understandingcited by 4
  5. 2024Contrastive Federated Learning with Tabular Data Siloscited by 2