mlsanityData to 7 Oct 2026

k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension

arXiv 2206.08526 ↗First posted 17 Jun 2022cs.IT math.IT stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Sliced mutual information (SMI) is defined as an average of mutual information (MI) terms between one-dimensional random projections of the random variables. It serves as a surrogate measure of dependence to classic MI that preserves many of its properties but is more scalable to high dimensions. However, a quantitative characterization of how SMI itself and estimation rates thereof depend on the ambient dimension, which is crucial to the understanding of scalability, remain obscure. This work provides a multifaceted account of the dependence of SMI on dimension, under a broader framework termed k-SMI, which considers projections to k-dimensional subspaces. Using a new result on the continuity of differential entropy in the 2-Wasserstein metric, we derive sharp bounds on the error of Monte Carlo (MC)-based estimates of k-SMI, with explicit dependence on k and the ambient dimension, revealing their interplay with the number of samples. We then combine the MC integrator with the neural estimation framework to provide an end-to-end k-SMI estimator, for which optimal convergence rates are established. We also explore asymptotics of the population k-SMI as dimension grows, providing Gaussian approximation results with a residual that decays under appropriate moment bounds. All our results trivially apply to SMI by setting k=1. Our theory is validated with numerical experiments and is applied to sliced InfoGAN, which altogether provide a comprehensive quantitative account of the scalability question of k-SMI, including SMI as a special case when k=1.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 53 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

21 of 53 references.

References · 21

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2016Deep Variational Information Bottleneckcited by 915
  3. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  4. 2018MINE: Mutual Information Neural Estimationcited by 680
  5. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  6. 2019Generalized Sliced Wasserstein Distancescited by 205
  7. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  8. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  9. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  10. 2020Statistical and Topological Properties of Sliced Probability Divergencescited by 83
  11. 2020Distributional Sliced-Wasserstein and Applications to Generative Modelingcited by 69
  12. 2019Sliced Gromov-Wassersteincited by 66
9 more
  1. 2020The Information Bottleneck Problem and Its Applications in Machine Learningcited by 66
  2. 2018Entropy and mutual information in models of deep neural networkscited by 50
  3. 2020On Projection Robust Optimal Transport: Sample Complexity and Model Misspecificationcited by 34
  4. 2021Fast Approximation of the Sliced-Wasserstein Distance Using Concentration of Random Projectionscited by 33
  5. 2021Neural Estimation of Statistical Divergencescited by 25
  6. 2021Non-Asymptotic Performance Guarantees for Neural Estimation of mathsff-Divergencescited by 15
  7. 2019Neural Entropic Estimation: A faster path to mutual information estimationcited by 9
  8. 2018Estimating Information Flow in Deep Neural Networkscited by 7
  9. 2021Statistical and Topological Properties of Gaussian Smoothed Sliced Probability Divergencescited by 7

Built on by

Not cited yet.