mlsanityData to 7 Oct 2026

Self-supervised Pretraining of Visual Features in the Wild

arXiv 2103.01988 ↗First posted 2 Mar 2021cs.CV cs.AIcited by 197

Not on the map. Why?

Abstract (arXiv)

Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 512 GPUs achieves 84.2% top-1 accuracy, surpassing the best self-supervised pretrained model by 1% and confirming that self-supervised learning works in a real world setting. Interestingly, we also observe that self-supervised models are good few-shot learners achieving 77.9% top-1 with access to only 10% of ImageNet. Code: https://github.com/facebookresearch/vissl

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Builds on 43 of its 58 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

43 of 58 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2017Mask R-CNNcited by 7,259
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
31 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  12. 2020Designing Network Design Spacescited by 954
  13. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  14. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  15. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  16. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  17. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  18. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  19. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  20. 2017Large Batch Training of Convolutional Networkscited by 529
  21. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  22. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  23. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  24. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  25. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  26. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  27. 2020Unsupervised pretraining transfers well across languagescited by 100
  28. 2017Unsupervised Learning by Predicting Noisecited by 97
  29. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  30. 2020Milking CowMask for Semi-Supervised Image Classificationcited by 35
  31. 2020Fixing the train-test resolution discrepancy: FixEfficientNetcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 58; 2023: 43; 2024: 33; 2025: 18; 2026: 4.

197 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  5. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  6. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  7. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  8. 2023A Cookbook of Self-Supervised Learningcited by 248
  9. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 246
  10. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  11. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  12. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
38 more
  1. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  2. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  3. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  4. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  5. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  6. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  7. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  8. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  9. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  10. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  11. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  12. 2021Learning to See by Looking at Noisecited by 92
  13. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  14. 2021Backdoor Attacks on Self-Supervised Learningcited by 89
  15. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  16. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  17. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  18. 2022Object discovery and representation networkscited by 80
  19. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  20. 2022Knowledge Tracing: A Surveycited by 76
  21. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72
  22. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  23. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  24. 2023A Billion-scale Foundation Model for Remote Sensing Imagescited by 62
  25. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  26. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  27. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  28. 2022Discovering Object Masks with Transformers for Unsupervised Semantic Segmentationcited by 54
  29. 2022Dissecting Self-Supervised Learning Methods for Surgical Computer Visioncited by 48
  30. 2024Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approachcited by 48
  31. 2021The Emergence of Objectness: Learning Zero-Shot Segmentation from Videoscited by 45
  32. 2022On Data Scaling in Masked Image Modelingcited by 45
  33. 2024BigGait: Learning Gait Representation You Want by Large Vision Modelscited by 44
  34. 2021Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Modelscited by 41
  35. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  36. 2022Robust and Efficient Medical Imaging with Self-Supervisioncited by 40
  37. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  38. 2021Contrasting Contrastive Self-Supervised Representation Learning Pipelinescited by 37

50 most cited of 197 All 197 (JSON)