# Justices for Information Bottleneck Theory

arXiv: https://arxiv.org/abs/2305.11387 · HTML: https://mlsanity.com/paper/2305.11387/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 9 of 9)

- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Unsupervised Speech Decomposition via Triple Information Bottleneck](https://mlsanity.com/paper/2004.11284/) (2020)
- [ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction](https://mlsanity.com/paper/2105.10446/) (2021)
- [Neuron Campaign for Initialization Guided by Information Bottleneck Theory](https://mlsanity.com/paper/2108.06530/) (2021)
- [Goal-Oriented Communication for Edge Learning based on the Information Bottleneck](https://mlsanity.com/paper/2202.12639/) (2022)
- [Efficient Maximal Coding Rate Reduction by Variational Forms](https://mlsanity.com/paper/2204.00077/) (2022)
- [Gaussian Lower Bound for the Information Bottleneck Limit](https://mlsanity.com/paper/1711.02421/) (2017)

## Abstract (arXiv)

This study comes as a timely response to mounting criticism of the information bottleneck (IB) theory, injecting fresh perspectives to rectify misconceptions and reaffirm its validity. Firstly, we introduce an auxiliary function to reinterpret the maximal coding rate reduction method as a special yet local optimal case of IB theory. Through this auxiliary function, we clarify the paradox of decreasing mutual information during the application of ReLU activation in deep learning (DL) networks. Secondly, we challenge the doubts about IB theory's applicability by demonstrating its capacity to explain the absence of a compression phase with linear activation functions in hidden layers, when viewed through the lens of the auxiliary function. Lastly, by taking a novel theoretical stance, we provide a new way to interpret the inner organizations of DL networks by using IB theory, aligning them with recent experimental evidence. Thus, this paper serves as an act of justice for IB theory, potentially reinvigorating its standing and application in DL and other fields such as communications and biomedical research.