mlsanityData to 7 Oct 2026

Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift

arXiv 2312.03318 ↗First posted 6 Dec 2023cs.LG cs.CV stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Self-training and contrastive learning have emerged as leading techniques for incorporating unlabeled data, both under distribution shift (unsupervised domain adaptation) and when it is absent (semi-supervised learning). However, despite the popularity and compatibility of these techniques, their efficacy in combination remains unexplored. In this paper, we undertake a systematic empirical investigation of this combination, finding that (i) in domain adaptation settings, self-training and contrastive learning offer significant complementary gains; and (ii) in semi-supervised learning settings, surprisingly, the benefits are not synergistic. Across eight distribution shift datasets (e.g., BREEDs, WILDS), we demonstrate that the combined method obtains 3--8% higher accuracy than either approach independently. We then theoretically analyze these techniques in a simplified model of distribution shift, demonstrating scenarios under which the features produced by contrastive learning can yield a good initialization for self-training to further amplify gains and achieve optimal performance, even when either method alone would fail.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 64 of its 101 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

64 of 101 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
38 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  4. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  6. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  8. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  9. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  10. 2020In Search of Lost Domain Generalizationcited by 1,042
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  13. 2013Domain Generalization via Invariant Feature Representationcited by 622
  14. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 609
  15. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  16. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  17. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  18. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  19. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
  20. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  21. 2017Functional Map of the Worldcited by 338
  22. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  23. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  24. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  25. 2021A Survey on Deep Semi-supervised Learningcited by 250
  26. 2018Learning from positive and unlabeled data: a surveycited by 218
  27. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  28. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  29. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  30. 2016Correlation Alignment for Unsupervised Domain Adaptationcited by 146
  31. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  32. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  33. 2020A Unified View of Label Shift Estimationcited by 119
  34. 2019Regularized Learning for Domain Adaptation under Label Shiftscited by 118
  35. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  36. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  37. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 85
  38. 2020Coping with Label Shift via Distributionally Robust Optimisationcited by 69

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 5.

5 papers.

  1. 2024A Survey of Deep Graph Learning under Distribution Shifts: from Graph Out-of-Distribution Generalization to Adaptationcited by 16
  2. 2024Dynamic Against Dynamic: An Open-set Self-learning Frameworkcited by 1
  3. 2024Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptationcited by 1
  4. 2024Resilience in Knowledge Graph Embeddingscited by 1
  5. 2024Contrastive Multiple Instance Learning for Weakly Supervised Person ReIDcited by 0