# Deep Sufficient Representation Learning via Mutual Information

arXiv: https://arxiv.org/abs/2207.10772 · HTML: https://mlsanity.com/paper/2207.10772/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 43 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Deep Neural Networks for Estimation and Inference](https://mlsanity.com/paper/1809.09953/) (2018)
- [Nonparametric independence testing via mutual information](https://mlsanity.com/paper/1711.06642/) (2017)
- [Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality](https://mlsanity.com/paper/1907.02177/) (2019)
- [Robust Generalization via α-Mutual Information](https://mlsanity.com/paper/2001.06399/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Graph Sufficiency Perspective for Neural Networks](https://mlsanity.com/paper/2507.10215/) (2025)
- [Sufficient Dimesion Reduction via Generalized Stein's Lemma](https://mlsanity.com/paper/2608.15121/) (2026)

## Abstract (arXiv)

We propose a mutual information-based sufficient representation learning (MSRL) approach, which uses the variational formulation of the mutual information and leverages the approximation power of deep neural networks. MSRL learns a sufficient representation with the maximum mutual information with the response and a user-selected distribution. It can easily handle multi-dimensional continuous or categorical response variables. MSRL is shown to be consistent in the sense that the conditional probability density function of the response variable given the learned representation converges to the conditional probability density function of the response variable given the predictor. Non-asymptotic error bounds for MSRL are also established under suitable conditions. To establish the error bounds, we derive a generalized Dudley's inequality for an order-two U-process indexed by deep neural networks, which may be of independent interest. We discuss how to determine the intrinsic dimension of the underlying data distribution. Moreover, we evaluate the performance of MSRL via extensive numerical experiments and real data analysis and demonstrate that MSRL outperforms some existing nonlinear sufficient dimension reduction methods.