# Nonlinear Information Bottleneck

arXiv: https://arxiv.org/abs/1705.02436 · HTML: https://mlsanity.com/paper/1705.02436/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 May 2017.

## Built on by (most cited first, 30 of 71)

- [Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off](https://mlsanity.com/paper/2209.09056/) (2022)
- [Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defenses](https://mlsanity.com/paper/2402.04013/) (2024)
- [PAC-Bayes Information Bottleneck](https://mlsanity.com/paper/2109.14509/) (2021)
- [Measuring Dependence with Matrix-based Entropy Functional](https://mlsanity.com/paper/2101.10160/) (2021)
- [Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional](https://mlsanity.com/paper/2102.00533/) (2021)
- [Mutual Information State Intrinsic Control](https://mlsanity.com/paper/2103.08107/) (2021)
- [IB-UQ: Information bottleneck based uncertainty quantification for neural function regression and neural operator learning](https://mlsanity.com/paper/2302.03271/) (2023)
- [Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration](https://mlsanity.com/paper/2103.12300/) (2021)
- [MemStream: Memory-Based Streaming Anomaly Detection](https://mlsanity.com/paper/2106.03837/) (2021)
- [Progressive Feedforward Collapse of ResNet Training](https://mlsanity.com/paper/2405.00985/) (2024)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [Token Communication in the Era of Large Models: An Information Bottleneck-Based Approach](https://mlsanity.com/paper/2507.01728/) (2025)
- [Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2312.09539/) (2023)
- [Information decomposition in complex systems via machine learning](https://mlsanity.com/paper/2307.04755/) (2023)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [R2-Trans:Fine-Grained Visual Categorization with Redundancy Reduction](https://mlsanity.com/paper/2204.10095/) (2022)
- [MvHo-IB: Multi-View Higher-Order Information Bottleneck for Brain Disorder Diagnosis](https://mlsanity.com/paper/2507.02847/) (2025)
- [Interpretability with full complexity by constraining feature information](https://mlsanity.com/paper/2211.17264/) (2022)
- [Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs](https://mlsanity.com/paper/2601.22709/) (2026)
- [Adversarial Information Bottleneck](https://mlsanity.com/paper/2103.00381/) (2021)
- [Multi-view Information Bottleneck Without Variational Approximation](https://mlsanity.com/paper/2204.10530/) (2022)
- [Recognizable Information Bottleneck](https://mlsanity.com/paper/2304.14618/) (2023)
- [Computationally Efficient Approximations for Matrix-based Renyi's Entropy](https://mlsanity.com/paper/2112.13720/) (2021)
- [A robust estimator of mutual information for deep learning interpretability](https://mlsanity.com/paper/2211.00024/) (2022)
- [Expression Syntax Information Bottleneck for Math Word Problems](https://mlsanity.com/paper/2310.15664/) (2023)
- [The Causal Information Bottleneck and Optimal Causal Variable Abstractions](https://mlsanity.com/paper/2410.00535/) (2024)
- [A Generalized Information Bottleneck Theory of Deep Learning](https://mlsanity.com/paper/2509.26327/) (2025)
- [Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning](https://mlsanity.com/paper/2510.20644/) (2025)
- [Gated Information Bottleneck for Generalization in Sequential Environments](https://mlsanity.com/paper/2110.06057/) (2021)
- [Conditional entropy minimization principle for learning domain invariant representation features](https://mlsanity.com/paper/2201.10460/) (2022)

## Abstract (arXiv)

Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed "bottleneck" random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete X and Y with small state spaces, and continuous X and Y with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous X and Y, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed "variational IB" method on several real-world datasets.