# Measuring Dependence with Matrix-based Entropy Functional

arXiv: https://arxiv.org/abs/2101.10160 · HTML: https://mlsanity.com/paper/2101.10160/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 82 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [Nonlinear Information Bottleneck](https://mlsanity.com/paper/1705.02436/) (2017)
- [Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle](https://mlsanity.com/paper/1802.09766/) (2018)
- [Robust Learning with the Hilbert-Schmidt Independence Criterion](https://mlsanity.com/paper/1910.00270/) (2019)
- [Large-Scale Kernel Methods for Independence Testing](https://mlsanity.com/paper/1606.07892/) (2016)
- [Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration](https://mlsanity.com/paper/1804.06537/) (2018)
- [The Randomized Dependence Coefficient](https://mlsanity.com/paper/1304.7717/) (2013)

## Built on by (most cited first, 19 of 19)

- [Understanding Neural Networks with Logarithm Determinant Entropy Estimator](https://mlsanity.com/paper/2105.03705/) (2021)
- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [MvHo-IB: Multi-View Higher-Order Information Bottleneck for Brain Disorder Diagnosis](https://mlsanity.com/paper/2507.02847/) (2025)
- [Optimal Randomized Approximations for Matrix based Renyi's Entropy](https://mlsanity.com/paper/2205.07426/) (2022)
- [Multi-view Information Bottleneck Without Variational Approximation](https://mlsanity.com/paper/2204.10530/) (2022)
- [Computationally Efficient Approximations for Matrix-based Renyi's Entropy](https://mlsanity.com/paper/2112.13720/) (2021)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [An Independence-promoting Loss for Music Generation with Language Models](https://mlsanity.com/paper/2406.02315/) (2024)
- [Gated Information Bottleneck for Generalization in Sequential Environments](https://mlsanity.com/paper/2110.06057/) (2021)
- [Deep Deterministic Independent Component Analysis for Hyperspectral Unmixing](https://mlsanity.com/paper/2202.02951/) (2022)
- [Discovering Common Information in Multi-view Data](https://mlsanity.com/paper/2406.15043/) (2024)
- [Measuring Statistical Dependencies via Maximum Norm and Characteristic Functions](https://mlsanity.com/paper/2208.07934/) (2022)
- [Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study](https://mlsanity.com/paper/2307.08572/) (2023)
- [Deep Deterministic Nonlinear ICA via Total Correlation Minimization with Matrix-Based Entropy Functional](https://mlsanity.com/paper/2601.00904/) (2025)
- [MOLE: MOdular Learning FramEwork via Mutual Information Maximization](https://mlsanity.com/paper/2308.07772/) (2023)
- [Improving Intrusion Detection with Domain-Invariant Representation Learning in Latent Space](https://mlsanity.com/paper/2312.17300/) (2023)
- [Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation](https://mlsanity.com/paper/2511.12491/) (2025)
- [Multimodal Functional Maximum Correlation for Emotion Recognition](https://mlsanity.com/paper/2512.23076/) (2025)
- [DIPHINE: Diffusion-based Phi-ID Neural Estimator](https://mlsanity.com/paper/2606.18997/) (2026)

## Abstract (arXiv)

Measuring the dependence of data plays a central role in statistics and machine learning. In this work, we summarize and generalize the main idea of existing information-theoretic dependence measures into a higher-level perspective by the Shearer's inequality. Based on our generalization, we then propose two measures, namely the matrix-based normalized total correlation (Tα^*) and the matrix-based normalized dual total correlation (Dα^*), to quantify the dependence of multiple variables in arbitrary dimensional space, without explicit estimation of the underlying data distributions. We show that our measures are differentiable and statistically more powerful than prevalent ones. We also show the impact of our measures in four different machine learning problems, namely the gene regulatory network inference, the robust machine learning under covariate shift and non-Gaussian noises, the subspace outlier detection, and the understanding of the learning dynamics of convolutional neural networks (CNNs), to demonstrate their utilities, advantages, as well as implications to those problems. Code of our dependence measure is available at: https://bit.ly/AAAI-dependence