Explicit Mutual Information Maximization for Self-Supervised Learning
Not on the map. Why?
Abstract (arXiv)
Recently, self-supervised learning (SSL) has been extensively studied. Theoretically, mutual information maximization (MIM) is an optimal criterion for SSL, with a strong theoretical foundation in information theory. However, it is difficult to directly apply MIM in SSL since the data distribution is not analytically available in applications. In practice, many existing methods can be viewed as approximate implementations of the MIM criterion. This work shows that, based on the invariance property of MI, explicit MI maximization can be applied to SSL under a generic distribution assumption, i.e., a relaxed condition of the data distribution. We further illustrate this by analyzing the generalized Gaussian distribution. Based on this result, we derive a loss function based on the MIM criterion using only second-order statistics. We implement the new loss for SSL and demonstrate its effectiveness via extensive experiments.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 7 Sep 2024.
Builds on
32 of 42 references.
References · 32
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2022Generative Adversarial Networkscited by 5,603
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
20 more
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
- 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
- 2018MINE: Mutual Information Neural Estimationcited by 680
- 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
- 2020A Survey on Contrastive Self-supervised Learningcited by 553
- 2019On Variational Bounds of Mutual Informationcited by 550
- 2014Deep metric learning using Triplet networkcited by 421
- 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 402
- 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
- 2020Whitening for Self-Supervised Representation Learningcited by 278
- 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
- 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 83
- 2013Equitability, mutual information, and the maximal information coefficientcited by 45
- 2022Self-Supervised Learning with an Information Maximization Criterioncited by 38
- 2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
- 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
- 2024HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphscited by 15
- 2022Adaptive Soft Contrastive Learningcited by 13
- 2024Self-Supervised Learning for Time Series: Contrastive or Generative?cited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.