mlsanityData to 14 Oct 2026

Collapse-Proof Non-Contrastive Self-Supervised Learning

arXiv 2410.04959 ↗First posted 7 Oct 2024cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We present a principled and simplified design of the projector and loss function for non-contrastive self-supervised learning based on hyperdimensional computing. We theoretically demonstrate that this design introduces an inductive bias that encourages representations to be simultaneously decorrelated and clustered, without explicitly enforcing these properties. This bias provably enhances generalization and suffices to avoid known training failure modes, such as representation, dimensional, cluster, and intracluster collapses. We validate our theoretical findings on image datasets, including SVHN, CIFAR-10, CIFAR-100, and ImageNet-100. Our approach effectively combines the strengths of feature decorrelation and cluster-based self-supervised learning methods, overcoming training failure modes while achieving strong generalization in clustering and linear classification tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 69 of its 81 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

69 of 81 references.

References · 69

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  10. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  11. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  12. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
38 more
  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  2. 2018MINE: Mutual Information Neural Estimationcited by 683
  3. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  4. 2019On Variational Bounds of Mutual Informationcited by 559
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  7. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  8. 2019On Mutual Information Maximization for Representation Learningcited by 316
  9. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  10. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  11. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  12. 2020Whitening for Self-Supervised Representation Learningcited by 279
  13. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  14. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  15. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  16. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  17. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  18. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  19. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  20. 2018Spreading vectors for similarity searchcited by 118
  21. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  22. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  23. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  24. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  25. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  26. 2021A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformationscited by 99
  27. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  28. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  29. 2021Equivariant Contrastive Learningcited by 66
  30. 2020S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generationcited by 63
  31. 2022The Hidden Uniform Cluster Prior in Self-Supervised Learningcited by 58
  32. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  33. 2022Learning Symmetric Embeddings for Equivariant World Modelscited by 54
  34. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  35. 2023Self-supervised learning of Split Invariant Equivariant representationscited by 44
  36. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  37. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 39
  38. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Dual Perspectives on Non-Contrastive Self-Supervised Learningcited by 4
  2. 2025Contrastive Self-Supervised Learning As Neural Manifold Packingcited by 3
  3. 2026Global Geometry Is Not Enough for Vision Representationscited by 2
  4. 2026Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architecturescited by 1
  5. 2025Why Prototypes Collapse: Diagnosing and Preventing Partial Collapse in Prototypical Self-Supervised Learningcited by 0
  6. 2025SARL: Spatially-Aware Self-Supervised Representation Learning for Visuo-Tactile Perceptioncited by 0
  7. 2026HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learningcited by 0