mlsanityData to 7 Oct 2026

Variance Covariance Regularization Enforces Pairwise Independence in Self-Supervised Representations

arXiv 2209.14905 ↗First posted 29 Sep 2022cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) methods such as VICReg, Barlow Twins or W-MSE avoid collapse of their joint embedding architectures by constraining or regularizing the covariance matrix of their projector's output. This study highlights important properties of such strategy, which we coin Variance-Covariance regularization (VCReg). More precisely, we show that em VCReg combined to a MLP projector enforces pairwise independence between the features of the learned representation. This result emerges by bridging VCReg applied on the projector's output to kernel independence criteria applied on the projector's input. We empirically validate our findings where (i) we put in evidence which projector's characteristics favor pairwise independence, (ii) we demonstrate pairwise independence to be beneficial for out-of-domain generalization, (iii) we demonstrate that the scope of VCReg goes beyond SSL by using it to solve Independent Component Analysis. This provides the first theoretical motivation and explanation of MLP projectors in SSL.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 41 of its 72 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

41 of 72 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  12. 2017Deep Setscited by 1,394
29 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  2. 2018Sanity Checks for Saliency Mapscited by 1,149
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  4. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  5. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  6. 2015Weight Uncertainty in Neural Networkscited by 554
  7. 2020Understanding the Behaviour of Contrastive Losscited by 397
  8. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  9. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  10. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  11. 2020Whitening for Self-Supervised Representation Learningcited by 278
  12. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  13. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  14. 2020Intriguing Properties of Contrastive Lossescited by 129
  15. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  16. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  17. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  18. 2020On Disentangled Representations Learned From Correlated Datacited by 94
  19. 2018Decorrelated Batch Normalizationcited by 83
  20. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  21. 2016Kernel-based Tests for Joint Independencecited by 40
  22. 2019Robust Learning with the Hilbert-Schmidt Independence Criterioncited by 36
  23. 2017Learning Independent Features with Adversarial Nets for Non-linear ICAcited by 33
  24. 2021A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learningcited by 26
  25. 2020Towards Good Practices in Self-supervised Representation Learningcited by 19
  26. 2022Toward a Geometrical Understanding of Self-supervised Contrastive Learningcited by 17
  27. 2019Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networkscited by 11
  28. 2019REMAP: Multi-layer entropy-guided pooling of dense CNN features for image retrievalcited by 9
  29. 2016Relative Error Embeddings for the Gaussian Kernel Distancecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 2; 2026: 2.

9 papers.

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2023Self-Supervised Learning with Lie Symmetries for Partial Differential Equationscited by 29
  3. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  4. 2026Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representationscited by 8
  5. 2024Graph Contrastive Invariant Learning from the Causal Perspectivecited by 6
  6. 2024Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximizationcited by 6
  7. 2023More From Less: Self-Supervised Knowledge Distillation for Routine Histopathology Datacited by 1
  8. 2025Adversarial Dependence Minimizationcited by 1
  9. 2026Transferable Graph Condensation from the Causal Perspectivecited by 0