mlsanityData to 14 Oct 2026

VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

arXiv 2105.04906 ↗First posted 11 May 2021cs.CV cs.AI cs.LGcited by 1,057

Not on the map. Why?

Cited by 72 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

Key facts. Cited by 1,057 papers in the map (data through 14 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 11 May 2021.

Builds on

43 of 66 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2017Mask R-CNNcited by 7,277
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
31 more
  1. 2016Wide Residual Networkscited by 3,070
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  11. 2019Contrastive Multiview Codingcited by 1,301
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  13. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  14. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  15. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  16. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
  17. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  18. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  19. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  20. 2017Large Batch Training of Convolutional Networkscited by 535
  21. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  22. 2020Whitening for Self-Supervised Representation Learningcited by 279
  23. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  24. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  25. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  26. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  27. 2020BYOL works even without batch statisticscited by 95
  28. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  29. 2019Unsupervised Deep Learning by Neighbourhood Discoverycited by 78
  30. 2020Learning Representations by Predicting Bags of Visual Wordscited by 73
  31. 2016CliqueCNN: Deep Unsupervised Exemplar Learningcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 40; 2022: 164; 2023: 231; 2024: 194; 2025: 136; 2026: 292.

1,057 papers.

Sort:

By area:

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  2. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  3. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  4. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  5. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 304
  6. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  7. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  8. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 273
  9. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  10. 2023A Cookbook of Self-Supervised Learningcited by 250
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
38 more
  1. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 169
  2. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  3. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 160
  4. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  5. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  6. 2021Self-Supervised Models are Continual Learnerscited by 149
  7. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  8. 2021Decoupled Contrastive Learningcited by 143
  9. 2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
  10. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  11. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  12. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  13. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  14. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  15. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  16. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  17. 2021An Empirical Study of Graph Contrastive Learningcited by 113
  18. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  19. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  20. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  21. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  22. 2023Large-scale Training of Foundation Models for Wearable Biosignalscited by 95
  23. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 89
  24. 2022Mind Reader: Reconstructing complex images from brain activitiescited by 85
  25. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  26. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  27. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  28. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  29. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
  30. 2022SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detectioncited by 74
  31. 2022Human alignment of neural network representationscited by 74
  32. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  33. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  34. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 72
  35. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  36. 2022Improving Test-Time Adaptation via Shift-agnostic Weight Regularization and Nearest Source Prototypescited by 67
  37. 2021High Fidelity Visualization of What Your Self-Supervised Representation Knows Aboutcited by 66
  38. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65

50 most cited of 1,057 All 1,057 (JSON)