# The Conditional Entropy Bottleneck

arXiv: https://arxiv.org/abs/2002.05379 · HTML: https://mlsanity.com/paper/2002.05379/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2020.

## Built on by (most cited first, 30 of 75)

- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Task-Oriented Communication for Multi-Device Cooperative Edge Inference](https://mlsanity.com/paper/2109.00172/) (2021)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning](https://mlsanity.com/paper/2106.04152/) (2021)
- [Compressive Visual Representations](https://mlsanity.com/paper/2109.12909/) (2021)
- [DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck](https://mlsanity.com/paper/2102.13268/) (2021)
- [Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness](https://mlsanity.com/paper/2106.02734/) (2021)
- [An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.01549/) (2024)
- [Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration](https://mlsanity.com/paper/2103.12300/) (2021)
- [An Empirical Investigation of Representation Learning for Imitation](https://mlsanity.com/paper/2205.07886/) (2022)
- [Balancing Multimodal Training Through Game-Theoretic Regularization](https://mlsanity.com/paper/2411.07335/) (2024)
- [An Information Criterion for Controlled Disentanglement of Multimodal Data](https://mlsanity.com/paper/2410.23996/) (2024)
- [Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Images](https://mlsanity.com/paper/2203.01810/) (2022)
- [Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility](https://mlsanity.com/paper/2207.04895/) (2022)
- [Pruning Adversarially Robust Neural Networks without Adversarial Examples](https://mlsanity.com/paper/2210.04311/) (2022)
- [Generalization in Quantum Machine Learning: a Quantum Information Perspective](https://mlsanity.com/paper/2102.08991/) (2021)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations](https://mlsanity.com/paper/2207.13224/) (2022)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow](https://mlsanity.com/paper/2502.20750/) (2025)
- [There Was Never a Bottleneck in Concept Bottleneck Models](https://mlsanity.com/paper/2506.04877/) (2025)
- [Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning](https://mlsanity.com/paper/2206.11396/) (2022)
- [Latent Representation and Simulation of Markov Processes via Time-Lagged Information Bottleneck](https://mlsanity.com/paper/2309.07200/) (2023)
- [Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models](https://mlsanity.com/paper/2502.11049/) (2025)
- [Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs](https://mlsanity.com/paper/2601.22709/) (2026)
- [Adversarial Information Bottleneck](https://mlsanity.com/paper/2103.00381/) (2021)
- [PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2210.08217/) (2022)
- [Multimodal Conditional Information Bottleneck for Generalizable AI-Generated Image Detection](https://mlsanity.com/paper/2505.15217/) (2025)

## Abstract (arXiv)

Much of the field of Machine Learning exhibits a prominent set of failure modes, including vulnerability to adversarial examples, poor out-of-distribution (OoD) detection, miscalibration, and willingness to memorize random labelings of datasets. We characterize these as failures of robust generalization, which extends the traditional measure of generalization as accuracy or related metrics on a held-out set. We hypothesize that these failures to robustly generalize are due to the learning systems retaining too much information about the training data. To test this hypothesis, we propose the Minimum Necessary Information (MNI) criterion for evaluating the quality of a model. In order to train models that perform well with respect to the MNI criterion, we present a new objective function, the Conditional Entropy Bottleneck (CEB), which is closely related to the Information Bottleneck (IB). We experimentally test our hypothesis by comparing the performance of CEB models with deterministic models and Variational Information Bottleneck (VIB) models on a variety of different datasets and robustness challenges. We find strong empirical evidence supporting our hypothesis that MNI models improve on these problems of robust generalization.