mlsanityData to 7 Oct 2026

Divide and Contrast: Self-supervised Learning from Uncurated Data

arXiv 2105.08054 ↗First posted 17 May 2021cs.CVcited by 85

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning holds promise in leveraging large amounts of unlabeled data, however much of its progress has thus far been limited to highly curated pre-training data such as ImageNet. We explore the effects of contrastive learning from larger, less-curated image datasets such as YFCC, and find there is indeed a large difference in the resulting representation quality. We hypothesize that this curation gap is due to a shift in the distribution of image classes -- which is more diverse and heavy-tailed -- resulting in less relevant negative samples to learn from. We test this hypothesis with a new approach, Divide and Contrast (DnC), which alternates between contrastive learning and clustering-based hard negative mining. When pretrained on less curated datasets, DnC greatly improves the performance of self-supervised learning on downstream tasks, while remaining competitive with the current state-of-the-art on curated datasets.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 95 of its 119 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

95 of 119 references.

References · 95

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2017Mask R-CNNcited by 7,259
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  13. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  14. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  15. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  16. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  17. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  18. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  19. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  20. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  21. 2019Contrastive Multiview Codingcited by 1,297
  22. 2016Colorful Image Colorizationcited by 1,113
  23. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  24. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  25. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  26. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  27. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  28. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  29. 2019Contrastive Representation Distillationcited by 734
  30. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  31. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  32. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  33. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  34. 2018Do Better ImageNet Models Transfer Better?cited by 560
  35. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  36. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  37. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  38. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 26; 2023: 22; 2024: 13; 2025: 11; 2026: 2.

85 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  3. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  4. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  5. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  6. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  7. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  8. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  9. 2021Self-Supervised Models are Continual Learnerscited by 149
  10. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  11. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  12. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
38 more
  1. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  2. 2022Object discovery and representation networkscited by 80
  3. 2024Continual Learning and Catastrophic Forgettingcited by 73
  4. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  5. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67
  6. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 57
  7. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 48
  8. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  9. 2023Towards In-context Scene Understandingcited by 45
  10. 2023Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Datacited by 44
  11. 2021SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Drivingcited by 43
  12. 2024RudolfV: A Foundation Model by Pathologists for Pathologistscited by 40
  13. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  14. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  15. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  16. 2021Object-aware Contrastive Learning for Debiased Scene Representationcited by 34
  17. 2022OpenCon: Open-world Contrastive Learningcited by 33
  18. 2022Indiscriminate Poisoning Attacks on Unsupervised Contrastive Learningcited by 32
  19. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  20. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  21. 2022UniVIP: A Unified Framework for Self-Supervised Visual Pre-trainingcited by 30
  22. 2023Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Imagescited by 30
  23. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  24. 2022Contrastive Learning with Boosted Memorizationcited by 26
  25. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  26. 2021Object-Aware Cropping for Self-Supervised Learningcited by 22
  27. 2022Self-supervised video pretraining yields robust and more human-aligned visual representationscited by 20
  28. 2022Location-Aware Self-Supervised Transformers for Semantic Segmentationcited by 20
  29. 2022Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routingcited by 18
  30. 2024What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insightscited by 13
  31. 2022Clustering Aided Weakly Supervised Training to Detect Anomalous Events in Surveillance Videoscited by 12
  32. 2023Coreset Sampling from Open-Set for Fine-Grained Self-Supervised Learningcited by 12
  33. 2021Separated Contrastive Learning for Organ-at-Risk and Gross-Tumor-Volume Segmentation with Limited Annotationcited by 11
  34. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11
  35. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  36. 2023RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Datacited by 10
  37. 2024Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulationcited by 10
  38. 2021A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clusteringcited by 9

50 most cited of 85 All 85 (JSON)