# On the Maximum Mutual Information Capacity of Neural Architectures

arXiv: https://arxiv.org/abs/2006.06037 · HTML: https://mlsanity.com/paper/2006.06037/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 2 of 2)

- [Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group](https://mlsanity.com/paper/2107.06898/) (2021)
- [Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone](https://mlsanity.com/paper/2609.23087/) (2026)

## Abstract (arXiv)

We derive the closed-form expression of the maximum mutual information - the maximum value of I(X;Z) obtainable via training - for a broad family of neural network architectures. The quantity is essential to several branches of machine learning theory and practice. Quantitatively, we show that the maximum mutual information for these families all stem from generalizations of a single catch-all formula. Qualitatively, we show that the maximum mutual information of an architecture is most strongly influenced by the width of the smallest layer of the network - the "information bottleneck" in a different sense of the phrase, and by any statistical invariances captured by the architecture.