mlsanityData to 7 Oct 2026

What Makes for Good Views for Contrastive Learning?

arXiv 2005.10243 ↗First posted 20 May 2020cs.CV cs.LGcited by 843

Not on the map. Why?

Abstract (arXiv)

Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning. Despite its success, the influence of different view choices has been less studied. In this paper, we use theoretical and empirical analysis to better understand the importance of view selection, and argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. To verify this hypothesis, we devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. We also consider data augmentation as a way to reduce MI, and show that increasing data augmentation indeed leads to decreasing MI and improves downstream classification accuracy. As a by-product, we achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification (73% top-1 linear readout with a ResNet-50). In addition, transferring our models to PASCAL VOC object detection and COCO instance segmentation consistently outperforms supervised pre-training. Code:http://github.com/HobbitLong/PyContrast

Key facts. Cited by 843 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 189; 2022: 213; 2023: 191; 2024: 117; 2025: 85; 2026: 48.

843 papers.

Sort:

By area:

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  5. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  6. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  7. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  8. 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
  9. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  10. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  11. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  12. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
38 more
  1. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  2. 2021Adversarial Graph Augmentation to Improve Graph Contrastive Learningcited by 200
  3. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  4. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  5. 2021Partial success in closing the gap between human and machine visioncited by 185
  6. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  7. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  8. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  9. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  10. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  11. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  12. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  13. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  14. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  15. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  16. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  17. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  18. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  19. 2021Decoupled Contrastive Learningcited by 141
  20. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  21. 2021Targeted Supervised Contrastive Learning for Long-Tailed Recognitioncited by 132
  22. 2023GEO-Bench: Toward Foundation Models for Earth Monitoringcited by 122
  23. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  24. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  25. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  26. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  27. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  28. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  29. 2025TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learningcited by 107
  30. 2021InfoGCL: Information-Aware Graph Contrastive Learningcited by 105
  31. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  32. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  33. 2021Contrastive Learning with Stronger Augmentationscited by 100
  34. 2021Instance Localization for Self-supervised Detection Pretrainingcited by 99
  35. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  36. 2021Large-scale Unsupervised Semantic Segmentationcited by 98
  37. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  38. 2021Region Similarity Representation Learningcited by 96

50 most cited of 843 All 843 (JSON)