mlsanityData to 7 Oct 2026

An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization

arXiv 2303.00633 ↗First posted 1 Mar 2023cs.IT cs.AI math.ITcited by 25

Not on the map. Why?

Abstract (arXiv)

Variance-Invariance-Covariance Regularization (VICReg) is a self-supervised learning (SSL) method that has shown promising results on a variety of tasks. However, the fundamental mechanisms underlying VICReg remain unexplored. In this paper, we present an information-theoretic perspective on the VICReg objective. We begin by deriving information-theoretic quantities for deterministic networks as an alternative to unrealistic stochastic network assumptions. We then relate the optimization of the VICReg objective to mutual information optimization, highlighting underlying assumptions and facilitating a constructive comparison with other SSL algorithms and derive a generalization bound for VICReg, revealing its inherent advantages for downstream tasks. Building on these results, we introduce a family of SSL methods derived from information-theoretic principles that outperform existing SSL techniques.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 57 of its 92 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

57 of 92 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
38 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  2. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  4. 2016Deep Variational Information Bottleneckcited by 915
  5. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  6. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  9. 2019On Variational Bounds of Mutual Informationcited by 550
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  12. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  13. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  14. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  15. 2017Implicit Regularization in Matrix Factorizationcited by 318
  16. 2019On Mutual Information Maximization for Representation Learningcited by 315
  17. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  18. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  19. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  20. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  21. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  22. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  23. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  24. 2017Generalization in Deep Learningcited by 146
  25. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  26. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  27. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  28. 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
  29. 2022Rethinking Minimal Sufficient Representation in Contrastive Learningcited by 67
  30. 2021Lossy Compression for Lossless Predictioncited by 51
  31. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  32. 2021Compressive Visual Representationscited by 41
  33. 2019Where is the Information in a Deep Neural Network?cited by 40
  34. 2017Estimating Mixture Entropy with Pairwise Distancescited by 34
  35. 2020Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networkscited by 28
  36. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  37. 2022Information Flow in Deep Neural Networkscited by 22
  38. 2018Eigendecomposition-free Training of Deep Networks with Zero Eigenvalue-based Lossescited by 15

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 12; 2025: 2; 2026: 3.

25 papers.

Sort:

By area:

  1. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  2. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  3. 2023Reverse Engineering Self-Supervised Learningcited by 33
  4. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  5. 2023Information Flow in Self-Supervised Learningcited by 16
  6. 2024Video Representation Learning with Joint-Embedding Predictive Architecturescited by 14
  7. 2023Variance-Covariance Regularization Improves Representation Learningcited by 13
  8. 2024Poly-View Contrastive Learningcited by 9
  9. 2023Self-Supervised Learning of Representations for Space Generates Multi-Modular Grid Cellscited by 8
  10. 2024Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representationscited by 8
  11. 2024Collapse-Proof Non-Contrastive Self-Supervised Learningcited by 7
  12. 2024Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximizationcited by 6
13 more
  1. 2024Tight PAC-Bayesian Risk Certificates for Contrastive Learningcited by 6
  2. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5
  3. 2026The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Modelscited by 2
  4. 2023Feature Learning in Image Hierarchies using Functional Maximal Correlationcited by 1
  5. 2024A Novel Transformer-Based Self-Supervised Learning Method to Enhance Photoplethysmogram Signal Artifact Detectioncited by 1
  6. 2024Explicit Mutual Information Maximization for Self-Supervised Learningcited by 1
  7. 2025The Alignment Bottleneckcited by 1
  8. 2023Unveiling the Potential of Probabilistic Embeddings in Self-Supervised Learningcited by 0
  9. 2023Improving Intrusion Detection with Domain-Invariant Representation Learning in Latent Spacecited by 0
  10. 2024Unsqueeze [CLS] Bottleneck to Learn Rich Representationscited by 0
  11. 2025Self-Supervised Learning with Gaussian Processescited by 0
  12. 2026A Multi-directional Meta-Learning Framework for Class-Generalizable Anomaly Detectioncited by 0
  13. 2026A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysiscited by 0