# On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views

arXiv: https://arxiv.org/abs/2002.00008 · HTML: https://mlsanity.com/paper/2002.00008/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2020.

## Built on by (most cited first, 28 of 28)

- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach](https://mlsanity.com/paper/2102.04170/) (2021)
- [Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional](https://mlsanity.com/paper/2102.00533/) (2021)
- [Semantic Information Recovery in Wireless Networks](https://mlsanity.com/paper/2204.13366/) (2022)
- [Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility](https://mlsanity.com/paper/2207.04895/) (2022)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [Information decomposition in complex systems via machine learning](https://mlsanity.com/paper/2307.04755/) (2023)
- [In-Network Learning: Distributed Training and Inference in Networks](https://mlsanity.com/paper/2107.03433/) (2021)
- [Lessons from Generalization Error Analysis of Federated Learning: You May Communicate Less Often!](https://mlsanity.com/paper/2306.05862/) (2023)
- [Information flows of diverse autoencoders](https://mlsanity.com/paper/2102.07402/) (2021)
- [Gated Information Bottleneck for Generalization in Sequential Environments](https://mlsanity.com/paper/2110.06057/) (2021)
- [On In-network learning. A Comparative Study with Federated and Split Learning](https://mlsanity.com/paper/2104.14929/) (2021)
- [Opening the Black Box of Deep Neural Networks in Physical Layer Communication](https://mlsanity.com/paper/2106.01124/) (2021)
- [Lossless Transformations and Excess Risk Bounds in Statistical Inference](https://mlsanity.com/paper/2307.16735/) (2023)
- [Statistically Valid Information Bottleneck via Multiple Hypothesis Testing](https://mlsanity.com/paper/2409.07325/) (2024)
- [Information-Bottleneck Driven Binary Neural Network for Change Detection](https://mlsanity.com/paper/2507.03504/) (2025)
- [Scalable Vector Gaussian Information Bottleneck](https://mlsanity.com/paper/2102.07525/) (2021)
- [Studying the Interplay between Information Loss and Operation Loss in Representations for Classification](https://mlsanity.com/paper/2112.15238/) (2021)
- [Theoretical Analysis of Deep Neural Networks in Physical Layer Communication](https://mlsanity.com/paper/2202.09954/) (2022)
- [Classification Utility, Fairness, and Compactness via Tunable Information Bottleneck and R'enyi Measures](https://mlsanity.com/paper/2206.10043/) (2022)
- [Elastic Information Bottleneck](https://mlsanity.com/paper/2311.03955/) (2023)
- [Deep Privacy Funnel Model: From a Discriminative to a Generative Approach with an Application to Face Recognition](https://mlsanity.com/paper/2404.02696/) (2024)
- [Multiview Representation Learning via Distributed Joint Latent Space Structuring](https://mlsanity.com/paper/2504.18455/) (2025)
- [Distributed Information Bottleneck Theory for Multi-Modal Task-Aware Semantic Communication](https://mlsanity.com/paper/2510.04000/) (2025)
- [On the Rate-Distortion-Complexity Tradeoff for Semantic Communication](https://mlsanity.com/paper/2602.14481/) (2026)
- [Source-Side Sufficiency for the Information Bottleneck: Exact Reduction and Finite-Block Equivalence](https://mlsanity.com/paper/2604.26744/) (2026)
- [The Information Bottleneck's Ordinary Differential Equation: First-Order Root-Tracking for the IB](https://mlsanity.com/paper/2306.09790/) (2023)
- [Semantic Rate-Distortion for Bounded Multi-Agent Communication: Capacity-Derived Semantic Spaces and the Communication Cost of Alignment](https://mlsanity.com/paper/2604.09521/) (2026)

## Abstract (arXiv)

This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate connections of this setting to remote source-coding under logarithmic loss distortion measure, information combining, common reconstruction, the Wyner-Ahlswede-Korner problem, the efficiency of investment information, as well as, generalization, variational inference, representation learning, autoencoders, and others are highlighted. We discuss its extension to the distributed information bottleneck problem with emphasis on the Gaussian model and highlight the basic connections to the uplink Cloud Radio Access Networks (CRAN) with oblivious processing. For this model, the optimal trade-offs between relevance (i.e., information) and complexity (i.e., rates) in the discrete and vector Gaussian frameworks is determined. In the concluding outlook, some interesting problems are mentioned such as the characterization of the optimal inputs ("features") distributions under power limitations maximizing the "relevance" for the Gaussian information bottleneck, under "complexity" constraints.