mlsanityData to 14 Oct 2026

Information Flow in Self-Supervised Learning

arXiv 2309.17281 ↗First posted 29 Sep 2023cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

In this paper, we conduct a comprehensive analysis of two dual-branch (Siamese architecture) self-supervised learning approaches, namely Barlow Twins and spectral contrastive learning, through the lens of matrix mutual information. We prove that the loss functions of these methods implicitly optimize both matrix mutual information and matrix joint entropy. This insight prompts us to further explore the category of single-branch algorithms, specifically MAE and U-MAE, for which mutual information and joint entropy become the entropy. Building on this intuition, we introduce the Matrix Variational Masked Auto-Encoder (M-MAE), a novel method that leverages the matrix-based estimation of entropy as a regularizer and subsumes U-MAE as a special case. The empirical evaluations underscore the effectiveness of M-MAE compared with the state-of-the-art methods, including a 3.9% improvement in linear probing ViT-Base, and a 1% improvement in fine-tuning ViT-Large, both on ImageNet.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 63 of its 69 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

63 of 69 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
38 more
  1. 2019Contrastive Multiview Codingcited by 1,301
  2. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  6. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  7. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  8. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  9. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  10. 2020Contrastive Learning with Hard Negative Samplescited by 628
  11. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  12. 2016Tutorial on Variational Autoencoderscited by 409
  13. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  14. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  15. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  16. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  17. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  18. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  19. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  20. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 142
  21. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  22. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  23. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  24. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  25. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  26. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  27. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  28. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  29. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  30. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54
  31. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  32. 2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
  33. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  34. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  35. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  36. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  37. 2022How to Understand Masked Autoencoderscited by 38
  38. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 5; 2026: 1.

16 papers.

Sort:

  1. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  2. 2024Provable Contrastive Continual Learningcited by 13
  3. 2023Understanding Grokking Through A Robustness Viewpointcited by 12
  4. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  5. 2023OTMatch: Improving Semi-Supervised Learning with Optimal Transportcited by 6
  6. 2024The Information of Large Language Model Geometrycited by 5
  7. 2025AdaDim: Dimensionality Adaptation for SSL Representational Dynamicscited by 4
  8. 2024Unveiling the Dynamics of Information Interplay in Supervised Learningcited by 3
  9. 2024Learning to Embed Distributions via Maximum Kernel Entropycited by 3
  10. 2024Large Language Model Evaluation via Matrix Nuclear-Normcited by 2
  11. 2024Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Trainingcited by 1
  12. 2023Deep Tensor Networkcited by 0
4 more
  1. 2025Information-Theoretic Perspectives on Optimizerscited by 0
  2. 2025Self-Supervised Representation Learning as Mutual Information Maximizationcited by 0
  3. 2025Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Directioncited by 0
  4. 2026Information theoretic underpinning of self-supervised learning by clusteringcited by 0