mlsanityData to 14 Oct 2026

Matrix Information Theory for Self-Supervised Learning

arXiv 2305.17326 ↗First posted 27 May 2023cs.LG cs.AI cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

The maximum entropy encoding framework provides a unified perspective for many non-contrastive learning methods like SimSiam, Barlow Twins, and MEC. Inspired by this framework, we introduce Matrix-SSL, a novel approach that leverages matrix information theory to interpret the maximum entropy encoding loss as matrix uniformity loss. Furthermore, Matrix-SSL enhances the maximum entropy encoding method by seamlessly incorporating matrix alignment loss, directly aligning covariance matrices in different branches. Experimental results reveal that Matrix-SSL outperforms state-of-the-art methods on the ImageNet dataset under linear evaluation settings and on MS-COCO for transfer learning tasks. Specifically, when performing transfer learning tasks on MS-COCO, our method outperforms previous SOTA methods such as MoCo v2 and BYOL up to 3.3% with only 400 epochs compared to 800 epochs pre-training. We also try to introduce representation learning into the language modeling regime by fine-tuning a 7B model using matrix cross-entropy loss, with a margin of 3.1% on the GSM8K dataset over the standard cross-entropy loss. Code available at https://github.com/yifanzhang-pro/Matrix-SSL.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 69 of its 94 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

69 of 94 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  6. 2019Contrastive Multiview Codingcited by 1,301
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  10. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  11. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  12. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  13. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  14. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  15. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  16. 2020Contrastive Learning with Hard Negative Samplescited by 628
  17. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  18. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  19. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  20. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  21. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  22. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 232
  23. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  24. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  25. 2018An efficient framework for learning sentence representationscited by 198
  26. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  27. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
  28. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  29. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  30. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  31. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  32. 2021On the Role of Neural Collapse in Transfer Learningcited by 100
  33. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  34. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  35. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  36. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  37. 2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 67
  38. 2020Contrastive estimation reveals topic posterior information to linear modelscited by 54

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 4; 2026: 4.

21 papers.

Sort:

  1. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  2. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  3. 2023Information Flow in Self-Supervised Learningcited by 16
  4. 2024Provable Contrastive Continual Learningcited by 13
  5. 2023Understanding Grokking Through A Robustness Viewpointcited by 12
  6. 2023OTMatch: Improving Semi-Supervised Learning with Optimal Transportcited by 6
  7. 2024The Information of Large Language Model Geometrycited by 5
  8. 2024Scaling Image Tokenizers with Grouped Spherical Quantizationcited by 4
  9. 2025AdaDim: Dimensionality Adaptation for SSL Representational Dynamicscited by 4
  10. 2024Unveiling the Dynamics of Information Interplay in Supervised Learningcited by 3
  11. 2023FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learningcited by 2
  12. 2024Exploiting Supervised Poison Vulnerability to Strengthen Self-Supervised Defensecited by 1
9 more
  1. 2024Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Trainingcited by 1
  2. 2025Countering Multi-modal Representation Collapse through Rank-targeted Fusioncited by 1
  3. 2026Vendi Novelty Scores for Out-of-Distribution Detectioncited by 1
  4. 2023Deep Tensor Networkcited by 0
  5. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
  6. 2025Information-Theoretic Perspectives on Optimizerscited by 0
  7. 2026Maximizing Incremental Information Entropy for Contrastive Learningcited by 0
  8. 2026Neural Collapse by Design: Learning Class Prototypes on the Hyperspherecited by 0
  9. 2026Efficient, Validation-Free Intrinsic Quality Estimation for Large-Scale Face Recognition Datasetscited by 0