mlsanityData to 14 Oct 2026

Establishing a stronger baseline for lightweight contrastive models

arXiv 2212.07158 ↗First posted 14 Dec 2022cs.CV cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent research has reported a performance degradation in self-supervised contrastive learning for specially designed efficient networks, such as MobileNet and EfficientNet. A common practice to address this problem is to introduce a pretrained contrastive teacher model and train the lightweight networks with distillation signals generated by the teacher. However, it is time and resource consuming to pretrain a teacher model when it is not available. In this work, we aim to establish a stronger baseline for lightweight contrastive models without using a pretrained teacher model. Specifically, we show that the optimal recipe for efficient models is different from that of larger models, and using the same training settings as ResNet50, as previous research does, is inappropriate. Additionally, we observe a common issu e in contrastive learning where either the positive or negative views can be noisy, and propose a smoothed version of InfoNCE loss to alleviate this problem. As a result, we successfully improve the linear evaluation results from 36.3% to 62.3% for MobileNet-V3-Large and from 42.2% to 65.8% for EfficientNet-B0 on ImageNet, closing the accuracy gap to ResNet50 with 5× fewer parameters. We hope our research will facilitate the usage of lightweight contrastive models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 24 of its 25 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

24 of 25 references.

References · 24

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  9. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  10. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  11. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  12. 2020Understanding the Behaviour of Contrastive Losscited by 398
12 more
  1. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  3. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  4. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  5. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  6. 2021Mean Shift for Self-Supervised Learningcited by 65
  7. 2021Incremental False Negative Detection for Contrastive Learningcited by 48
  8. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  9. 2021DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learningcited by 26
  10. 2021Temperature as Uncertainty in Contrastive Learningcited by 25
  11. 2021Bag of Instances Aggregation Boosts Self-supervised Distillationcited by 22
  12. 2021On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signalscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.

3 papers.

  1. 2023From Pretext to Purpose: Batch-Adaptive Self-Supervised Learningcited by 1
  2. 2025Contrastive Self-Supervised Learning at the Edge: An Energy Perspectivecited by 0
  3. 2026Maximizing Incremental Information Entropy for Contrastive Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Slimmable Networks for Contrastive Self-supervised Learning
  2. 2024On the Surprising Efficacy of Distillation as an Alternative to Pre-Training Small Models
  3. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
  4. 2021Boosting Contrastive Learning with Relation Knowledge Distillation
  5. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Models
  6. 2022Feature-domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution
  7. 2024Contrastive Learning with Synthetic Positives
  8. 2024Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures