mlsanityData to 14 Oct 2026

Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence

arXiv 2506.04053 ↗First posted 4 Jun 2025cs.LG cs.IT math.ITcited by 1

Not on the map. Why?

Abstract (arXiv)

Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence, prioritizes redundancy over informative content, and in some cases, performs worse than correlation coefficient.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

20 of 43 references.

References · 20

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  3. 2018Deep Graph Infomaxcited by 1,106
  4. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  5. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  6. 2018MINE: Mutual Information Neural Estimationcited by 683
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  8. 2019On Mutual Information Maximization for Representation Learningcited by 316
  9. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  10. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  11. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  12. 2023Beyond Normal: On the Evaluation of Mutual Information Estimatorscited by 37
8 more
  1. 2024Approximating mutual information of high-dimensional variables using learned representationscited by 21
  2. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compressioncited by 19
  3. 2023Max-Sliced Mutual Informationcited by 10
  4. 2022Conditional Independence Testing via Latent Representation Learningcited by 9
  5. 2023High-Dimensional Smoothed Entropy Estimation via Dimensionality Reductioncited by 7
  6. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  7. 2024Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMaxcited by 4
  8. 2024Leveraging Superfluous Information in Contrastive Representation Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025FMMI: Flow Matching Mutual Information Estimationcited by 4