# DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies

arXiv: https://arxiv.org/abs/2301.08164 · HTML: https://mlsanity.com/paper/2301.08164/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 19 Jan 2023.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://mlsanity.com/paper/1705.08584/) (2017)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [On Deep Multi-View Representation Learning: Objectives and Optimization](https://mlsanity.com/paper/1602.01024/) (2016)
- [Self-Supervised Learning with Kernel Dependence Maximization](https://mlsanity.com/paper/2106.08320/) (2021)
- [Image-to-image translation for cross-domain disentanglement](https://mlsanity.com/paper/1805.09730/) (2018)
- [Learning Disentangled Representations via Mutual Information Estimation](https://mlsanity.com/paper/1912.03915/) (2019)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [Rethinking InfoNCE: How Many Negative Samples Do You Need?](https://mlsanity.com/paper/2105.13003/) (2021)
- [Information Theoretic Learning with Infinitely Divisible Kernels](https://mlsanity.com/paper/1301.3551/) (2013)
- [Multi-view Information Bottleneck Without Variational Approximation](https://mlsanity.com/paper/2204.10530/) (2022)
- [The Representation Jensen-R'enyi Divergence](https://mlsanity.com/paper/2112.01583/) (2021)

## Built on by (most cited first, 21 of 21)

- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models](https://mlsanity.com/paper/2401.17139/) (2024)
- [Does Representation Matter? Exploring Intermediate Layers in Large Language Models](https://mlsanity.com/paper/2412.09563/) (2024)
- [Information Flow in Self-Supervised Learning](https://mlsanity.com/paper/2309.17281/) (2023)
- [UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective](https://mlsanity.com/paper/2410.03090/) (2024)
- [Understanding Grokking Through A Robustness Viewpoint](https://mlsanity.com/paper/2311.06597/) (2023)
- [Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning](https://mlsanity.com/paper/2411.02344/) (2024)
- [Enhancing Cross-task Transfer of Large Language Models via Activation Steering](https://mlsanity.com/paper/2507.13236/) (2025)
- [Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage](https://mlsanity.com/paper/2510.17833/) (2025)
- [Unveiling the Dynamics of Information Interplay in Supervised Learning](https://mlsanity.com/paper/2406.03999/) (2024)
- [Better, Faster: Harnessing Self-Improvement in Large Reasoning Models](https://mlsanity.com/paper/2605.24998/) (2026)
- [FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning](https://mlsanity.com/paper/2310.02903/) (2023)
- [Large Language Model Evaluation via Matrix Nuclear-Norm](https://mlsanity.com/paper/2410.10672/) (2024)
- [Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training](https://mlsanity.com/paper/2409.16767/) (2024)
- [Memorization-Compression Cycles Improve Generalization](https://mlsanity.com/paper/2505.08727/) (2025)
- [Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Models](https://mlsanity.com/paper/2604.10949/) (2026)
- [Generative Archetype-Grounded Item Representations for Sequential Recommendation](https://mlsanity.com/paper/2606.11023/) (2026)
- [Information-Theoretic Perspectives on Optimizers](https://mlsanity.com/paper/2502.20763/) (2025)
- [Seeing Before Answering: Training-Free Visual Layer Profiling for Vision-Language Models](https://mlsanity.com/paper/2608.16263/) (2026)
- [The Layer Mystery of VLA: An Information-Theoretical Analysis of VLA Latent Interface](https://mlsanity.com/paper/2609.36118/) (2026)
- [Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding](https://mlsanity.com/paper/2610.12939/) (2026)

## Abstract (arXiv)

We introduce an information-theoretic quantity with similar properties to mutual information that can be estimated from data without making explicit assumptions on the underlying distribution. This quantity is based on a recently proposed matrix-based entropy that uses the eigenvalues of a normalized Gram matrix to compute an estimate of the eigenvalues of an uncentered covariance operator in a reproducing kernel Hilbert space. We show that a difference of matrix-based entropies (DiME) is well suited for problems involving the maximization of mutual information between random variables. While many methods for such tasks can lead to trivial solutions, DiME naturally penalizes such outcomes. We compare DiME to several baseline estimators of mutual information on a toy Gaussian dataset. We provide examples of use cases for DiME, such as latent factor disentanglement and a multiview representation learning problem where DiME is used to learn a shared representation among views with high mutual information.