mlsanityData to 7 Oct 2026

Enhancing Contrastive Learning with Efficient Combinatorial Positive Pairing

arXiv 2401.05730 ↗First posted 11 Jan 2024cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In the past few years, contrastive learning has played a central role for the success of visual unsupervised representation learning. Around the same time, high-performance non-contrastive learning methods have been developed as well. While most of the works utilize only two views, we carefully review the existing multi-view methods and propose a general multi-view strategy that can improve learning speed and performance of any contrastive or non-contrastive method. We first analyze CMC's full-graph paradigm and empirically show that the learning speed of K-views can be increased by KC2 times for small learning rate and early training. Then, we upgrade CMC's full-graph by mixing views created by a crop-only augmentation, adopting small-size views as in SwAV multi-crop, and modifying the negative sampling. The resulting multi-view strategy is called ECPP (Efficient Combinatorial Positive Pairing). We investigate the effectiveness of ECPP by applying it to SimCLR and assessing the linear evaluation performance for CIFAR-10 and ImageNet-100. For each benchmark, we achieve a state-of-the-art performance. In case of ImageNet-100, ECPP boosted SimCLR outperforms supervised learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 54 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

48 of 54 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  12. 2018Group Normalizationcited by 1,514
36 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2019Contrastive Multiview Codingcited by 1,297
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  10. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  11. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  12. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  13. 2020Contrastive Learning with Hard Negative Samplescited by 619
  14. 2017Large Batch Training of Convolutional Networkscited by 529
  15. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  16. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  17. 2020Understanding the Behaviour of Contrastive Losscited by 397
  18. 2020Debiased Contrastive Learningcited by 354
  19. 2019On Mutual Information Maximization for Representation Learningcited by 315
  20. 2018Context Encoding for Semantic Segmentationcited by 289
  21. 2017Sampling Matters in Deep Embedding Learningcited by 285
  22. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  23. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  24. 2019Fixing the train-test resolution discrepancycited by 209
  25. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  26. 2018i-RevNet: Deep Invertible Networkscited by 137
  27. 2020Intriguing Properties of Contrastive Lossescited by 129
  28. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  29. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  30. 2017Smart Mining for Deep Metric Learningcited by 96
  31. 2020Hard negative examples are hard, but usefulcited by 82
  32. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  33. 2020Conditional Negative Sampling for Contrastive Learning of Visual Representationscited by 57
  34. 2019Hardness-Aware Deep Metric Learningcited by 55
  35. 2018Mining on Manifolds: Metric Learning without Labelscited by 36
  36. 2021Contrastive Attraction and Contrastive Repulsion for Representation Learningcited by 13

Built on by

Not cited yet.