mlsanityData to 14 Oct 2026

Accelerating Self-Supervised Learning via Efficient Training Strategies

arXiv 2212.05611 ↗First posted 11 Dec 2022cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Recently the focus of the computer vision community has shifted from expensive supervised learning towards self-supervised learning of visual representations. While the performance gap between supervised and self-supervised has been narrowing, the time for training self-supervised deep networks remains an order of magnitude larger than its supervised counterparts, which hinders progress, imposes carbon cost, and limits societal benefits to institutions with substantial resources. Motivated by these issues, this paper investigates reducing the training time of recent self-supervised methods by various model-agnostic strategies that have not been used for this problem. In particular, we study three strategies: an extendable cyclic learning rate schedule, a matching progressive augmentation magnitude and image resolutions schedule, and a hard positive mining strategy based on augmentation difficulty. We show that all three methods combined lead up to 2.7 times speed-up in the training time of several self-supervised methods while retaining comparable performance to the standard self-supervised learning setting.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 11 Dec 2022.

Builds on

42 of 50 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2013Auto-Encoding Variational Bayescited by 4,053
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
30 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  6. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  9. 2019Contrastive Multiview Codingcited by 1,301
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  11. 2016Colorful Image Colorizationcited by 1,114
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  13. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  14. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  15. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  16. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  17. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  18. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  19. 2017Three Factors Influencing Minima in SGDcited by 273
  20. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 226
  21. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  22. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  23. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 199
  24. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  25. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  26. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  27. 2020PyTorch Distributed: Experiences on Accelerating Data Parallel Trainingcited by 90
  28. 2020Fixing the train-test resolution discrepancy: FixEfficientNetcited by 31
  29. 2020MaxUp: A Simple Way to Improve Generalization of Neural Network Trainingcited by 27
  30. 2019Mix & Match: training convnets with mixed image sizes for improved accuracy, speed and scale resiliencycited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  2. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  3. 2024Accelerating Augmentation Invariance Pretrainingcited by 1
  4. 2025Progressive Growing of Patch Size: Curriculum Learning for Accelerated and Improved Medical Image Segmentationcited by 1
  5. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0