mlsanityData to 7 Oct 2026

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

arXiv 2006.09882 ↗First posted 17 Jun 2020cs.CVcited by 2,845

Not on the map. Why?

Cited by 56 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or views) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a swapped prediction mechanism where we predict the cluster assignment of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements much. We validate our findings by achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as surpassing supervised pretraining on all the considered transfer tasks.

Key facts. Cited by 2,845 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 458; 2022: 646; 2023: 674; 2024: 516; 2025: 366; 2026: 185.

2,845 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  10. 2022Diffusion Models in Vision: A Surveycited by 1,008
  11. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  12. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
38 more
  1. 2021Robust fine-tuning of zero-shot modelscited by 785
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  4. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  5. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  6. 2021Extract Free Dense Labels from CLIPcited by 580
  7. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  8. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  9. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  10. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  11. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  12. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  13. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  14. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  15. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 402
  16. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  17. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  18. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  19. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  20. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  21. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  22. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  23. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  24. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  25. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  26. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  27. 2021Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentationcited by 268
  28. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  29. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  30. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  31. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  32. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  33. 2022Visual Prompting via Image Inpaintingcited by 248
  34. 2023A Cookbook of Self-Supervised Learningcited by 248
  35. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  36. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  37. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  38. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222

50 most cited of 2,845 All 2,845 (JSON)