mlsanityData to 7 Oct 2026

The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning

arXiv 2307.10907 ↗First posted 20 Jul 2023cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

The mechanisms behind the success of multi-view self-supervised learning (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been studied through the lens of InfoNCE, a lower bound of the Mutual Information (MI). However, the relation between other MVSSL methods and MI remains unclear. We consider a different lower bound on the MI consisting of an entropy and a reconstruction term (ER), and analyze the main MVSSL families through its lens. Through this ER bound, we show that clustering-based methods such as DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of distillation-based approaches such as BYOL and DINO, showing that they explicitly maximize the reconstruction term and implicitly encourage a stable entropy, and we confirm this empirically. We show that replacing the objectives of common MVSSL methods with this ER bound achieves competitive performance, while making them stable when training with smaller batch sizes or smaller exponential moving average (EMA) coefficients. Github repo: https://github.com/apple/ml-entropy-reconstruction.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

38 of 53 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
26 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2018Deep Graph Infomaxcited by 1,101
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  8. 2016Deep Variational Information Bottleneckcited by 915
  9. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  10. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  11. 2018MINE: Mutual Information Neural Estimationcited by 680
  12. 2019On Variational Bounds of Mutual Informationcited by 550
  13. 2017Large Batch Training of Convolutional Networkscited by 529
  14. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  15. 2019On Mutual Information Maximization for Representation Learningcited by 315
  16. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  17. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  18. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  19. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  20. 2020The Information Bottleneck Problem and Its Applications in Machine Learningcited by 66
  21. 2020Conditional Negative Sampling for Contrastive Learning of Visual Representationscited by 57
  22. 2022Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performancecited by 40
  23. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 25
  24. 2021Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCEcited by 22
  25. 2019The Convex Information Bottleneck Lagrangiancited by 16
  26. 2021Stochastic Contrastive Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025Information-Guided Diffusion Sampling for Dataset Distillationcited by 10
  2. 2024Poly-View Contrastive Learningcited by 9
  3. 2024Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representationscited by 8
  4. 2024Separating common from salient patterns with Contrastive Representation Learningcited by 6
  5. 2026Understanding Self-Supervised Learning via Latent Distribution Matchingcited by 4
  6. 2025Generalized Category Discovery via Token Manifold Capacity Learningcited by 3
  7. 2024A Coding-Theoretic Analysis of Hyperspherical Prototypical Learning Geometrycited by 1
  8. 2025Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Datacited by 1
  9. 2025Multimodal Datasets with Controllable Mutual Informationcited by 1
  10. 2026MAPLE: Self-Supervised Learning-Enhanced Nonlinear Dimensionality Reduction for Visual Analysiscited by 1
  11. 2024Maximum Manifold Capacity Representations in State Representation Learningcited by 0
  12. 2026Provable Subspace Identification of Nonlinear Multi-view CCAcited by 0
1 more
  1. 2026Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisancecited by 0