mlsanityData to 14 Oct 2026

The Hidden Uniform Cluster Prior in Self-Supervised Learning

arXiv 2210.07277 ↗First posted 13 Oct 2022cs.LG cs.AI cs.CVcited by 58

Not on the map. Why?

Abstract (arXiv)

A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e.g., SimCLR, VICReg, SwAV, MSN). We show that in the formulation of all these methods is an overlooked prior to learn features that enable uniform clustering of the data. While this prior has led to remarkably semantic representations when pretraining on class-balanced data, such as ImageNet, we demonstrate that it can hamper performance when pretraining on class-imbalanced data. By moving away from conventional uniformity priors and instead preferring power-law distributed feature clusters, we show that one can improve the quality of the learned representations on real-world class-imbalanced datasets. To demonstrate this, we develop an extension of the Masked Siamese Networks (MSN) method to support the use of arbitrary features priors.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 53 of its 84 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

53 of 84 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  13. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  14. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  15. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  16. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  17. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  18. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  20. 2017Large Batch Training of Convolutional Networkscited by 535
  21. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  22. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  23. 2019On Mutual Information Maximization for Representation Learningcited by 316
  24. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  25. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  26. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  27. 2017Learning Discrete Representations via Information Maximizing Self-Augmented Trainingcited by 149
  28. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  29. 2020Intriguing Properties of Contrastive Lossescited by 130
  30. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  31. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  32. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  33. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  34. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  35. 2020Learning Representations by Predicting Bags of Visual Wordscited by 73
  36. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 66
  37. 2019Fair k-Center Clustering for Data Summarizationcited by 65
  38. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 20; 2024: 18; 2025: 13; 2026: 6.

58 papers.

Sort:

By area:

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  3. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  4. 2023A Cookbook of Self-Supervised Learningcited by 250
  5. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 166
  6. 2023Virchow: A Million-Slide Digital Pathology Foundation Modelcited by 90
  7. 2024InfoNCE: Identifying the Gap Between Theory and Practicecited by 40
  8. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  9. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  10. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  11. 2023Group Robust Classification Without Any Group Informationcited by 25
  12. 2024UniCorn: A Unified Contrastive Learning Approach for Multi-view Molecular Representation Learningcited by 18
38 more
  1. 2023Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representationscited by 16
  2. 2023Contrastive Tuning: A Little Help to Make Masked Autoencoders Forgetcited by 15
  3. 20243D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learningcited by 14
  4. 2024Self-Supervised Pre-Training with Joint-Embedding Predictive Architecture Boosts ECG Classification Performancecited by 14
  5. 2024What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insightscited by 13
  6. 2023Stochastic positional embeddings improve masked image modelingcited by 12
  7. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11
  8. 2023Unsupervised Embedding Quality Evaluationcited by 9
  9. 2024Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learningcited by 9
  10. 2025Clustering Properties of Self-Supervised Learningcited by 8
  11. 2024MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentationcited by 7
  12. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  13. 2025Improve Representation for Imbalanced Regression through Geometric Constraintscited by 7
  14. 2023Cross-view and Cross-pose Completion for 3D Human Understandingcited by 6
  15. 2024Beyond [cls]: Exploring the true potential of Masked Image Modeling representationscited by 6
  16. 2024On Partial Prototype Collapse in the DINO Family of Self-Supervised Methodscited by 5
  17. 2025A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasetscited by 5
  18. 2023A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representationcited by 4
  19. 2024S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Drivingcited by 4
  20. 2025Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learningcited by 4
  21. 2025From Video to EEG: Adapting Joint Embedding Predictive Architecture to Uncover Saptiotemporal Dynamics in Brain Signal Analysiscited by 4
  22. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  23. 2025Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculumcited by 3
  24. 2025Semantic Concentration for Self-Supervised Dense Representations Learningcited by 3
  25. 2023Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervisioncited by 2
  26. 2023An Investigation of Representation and Allocation Harms in Contrastive Learningcited by 2
  27. 2023Unifying Self-Supervised Clustering and Energy-Based Modelscited by 2
  28. 2024On Discriminative Probabilistic Modeling for Self-Supervised Representation Learningcited by 2
  29. 2023Benchmarking Low-Shot Robustness to Natural Distribution Shiftscited by 1
  30. 2024Self-supervised Pretraining and Finetuning for Monocular Depth and Visual Odometrycited by 1
  31. 2024Boosting Adverse Weather Crowd Counting via Multi-queue Contrastive Learningcited by 1
  32. 2024Self-supervised learning for radio-astronomy source classification: a benchmarkcited by 1
  33. 2026IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learningcited by 1
  34. 2026A Self supervised learning framework for imbalanced medical imaging datasetscited by 1
  35. 2023Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learningcited by 0
  36. 2023Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representationscited by 0
  37. 2024Rethinking Open-World Semi-Supervised Learning: Distribution Mismatch and Inductive Inferencecited by 0
  38. 2024Self-Supervised Anomaly Detection in the Wild: Favor Joint Embeddings Methodscited by 0

50 most cited of 58 All 58 (JSON)