mlsanityData to 7 Oct 2026

DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies

arXiv 2301.08164 ↗First posted 19 Jan 2023cs.LG cs.IT math.ITcited by 20

Not on the map. Why?

Abstract (arXiv)

We introduce an information-theoretic quantity with similar properties to mutual information that can be estimated from data without making explicit assumptions on the underlying distribution. This quantity is based on a recently proposed matrix-based entropy that uses the eigenvalues of a normalized Gram matrix to compute an estimate of the eigenvalues of an uncentered covariance operator in a reproducing kernel Hilbert space. We show that a difference of matrix-based entropies (DiME) is well suited for problems involving the maximization of mutual information between random variables. While many methods for such tasks can lead to trivial solutions, DiME naturally penalizes such outcomes. We compare DiME to several baseline estimators of mutual information on a toy Gaussian dataset. We provide examples of use cases for DiME, such as latent factor disentanglement and a multiview representation learning problem where DiME is used to learn a shared representation among views with high mutual information.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 25 of its 48 references that are in the map. First posted on arXiv 19 Jan 2023.

Builds on

25 of 48 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2019Contrastive Multiview Codingcited by 1,297
  6. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  7. 2019An Introduction to Variational Autoencoderscited by 997
  8. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  9. 2018Disentangling by Factorisingcited by 614
  10. 2019On Variational Bounds of Mutual Informationcited by 550
  11. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  12. 2019On Mutual Information Maximization for Representation Learningcited by 315
13 more
  1. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  2. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 251
  3. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 233
  4. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  5. 2016On Deep Multi-View Representation Learning: Objectives and Optimizationcited by 162
  6. 2021Self-Supervised Learning with Kernel Dependence Maximizationcited by 77
  7. 2018Image-to-image translation for cross-domain disentanglementcited by 61
  8. 2019Learning Disentangled Representations via Mutual Information Estimationcited by 52
  9. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 24
  10. 2021Rethinking InfoNCE: How Many Negative Samples Do You Need?cited by 18
  11. 2013Information Theoretic Learning with Infinitely Divisible Kernelscited by 9
  12. 2022Multi-view Information Bottleneck Without Variational Approximationcited by 5
  13. 2021The Representation Jensen-R'enyi Divergencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 5; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  2. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  3. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 31
  4. 2023Information Flow in Self-Supervised Learningcited by 16
  5. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 14
  6. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
  7. 2024Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoningcited by 9
  8. 2025Enhancing Cross-task Transfer of Large Language Models via Activation Steeringcited by 6
  9. 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantagecited by 4
  10. 2024Unveiling the Dynamics of Information Interplay in Supervised Learningcited by 3
  11. 2023FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learningcited by 2
  12. 2024Large Language Model Evaluation via Matrix Nuclear-Normcited by 2
8 more
  1. 2026Better, Faster: Harnessing Self-Improvement in Large Reasoning Modelscited by 2
  2. 2024Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Trainingcited by 1
  3. 2025Memorization-Compression Cycles Improve Generalizationcited by 1
  4. 2026Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Modelscited by 1
  5. 2026Generative Archetype-Grounded Item Representations for Sequential Recommendationcited by 1
  6. 2025Information-Theoretic Perspectives on Optimizerscited by 0
  7. 2026Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Modelscited by 0
  8. 2026The Layer Mystery of VLA: An Information-Theoretical Analysis of VLA Latent Interfacecited by 0