# Elastic Information Bottleneck

arXiv: https://arxiv.org/abs/2311.03955 · HTML: https://mlsanity.com/paper/2311.03955/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 7 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Graph Information Bottleneck](https://mlsanity.com/paper/2010.12811/) (2020)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [How much does your data exploration overfit? Controlling bias via information usage](https://mlsanity.com/paper/1511.05219/) (2015)
- [InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective](https://mlsanity.com/paper/2010.02329/) (2020)
- [The Conditional Entropy Bottleneck](https://mlsanity.com/paper/2002.05379/) (2020)
- [The Information Bottleneck Problem and Its Applications in Machine Learning](https://mlsanity.com/paper/2004.14941/) (2020)
- [The deterministic information bottleneck](https://mlsanity.com/paper/1604.00268/) (2016)
- [On Learning Invariant Representation for Domain Adaptation](https://mlsanity.com/paper/1901.09453/) (2019)
- [Caveats for information bottleneck in deterministic scenarios](https://mlsanity.com/paper/1808.07593/) (2018)
- [Learning Optimal Representations with the Decodable Information Bottleneck](https://mlsanity.com/paper/2009.12789/) (2020)
- [Distributed Variational Representation Learning](https://mlsanity.com/paper/1807.04193/) (2018)
- [On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views](https://mlsanity.com/paper/2002.00008/) (2020)
- [PAC-Bayes Information Bottleneck](https://mlsanity.com/paper/2109.14509/) (2021)
- [Discriminative Feature Alignment: Improving Transferability of Unsupervised Domain Adaptation by Gaussian-guided Latent Alignment](https://mlsanity.com/paper/2006.12770/) (2020)
- [Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithms](https://mlsanity.com/paper/2203.02474/) (2022)
- [Multivariate Information Bottleneck](https://mlsanity.com/paper/1301.2270/) (2013)
- [Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning](https://mlsanity.com/paper/2206.02604/) (2022)

## Built on by (most cited first, 1 of 1)

- [An Information Bottleneck Asset Pricing Model](https://mlsanity.com/paper/2507.23218/) (2025)

## Abstract (arXiv)

Information bottleneck is an information-theoretic principle of representation learning that aims to learn a maximally compressed representation that preserves as much information about labels as possible. Under this principle, two different methods have been proposed, i.e., information bottleneck (IB) and deterministic information bottleneck (DIB), and have gained significant progress in explaining the representation mechanisms of deep learning algorithms. However, these theoretical and empirical successes are only valid with the assumption that training and test data are drawn from the same distribution, which is clearly not satisfied in many real-world applications. In this paper, we study their generalization abilities within a transfer learning scenario, where the target error could be decomposed into three components, i.e., source empirical error, source generalization gap (SG), and representation discrepancy (RD). Comparing IB and DIB on these terms, we prove that DIB's SG bound is tighter than IB's while DIB's RD is larger than IB's. Therefore, it is difficult to tell which one is better. To balance the trade-off between SG and the RD, we propose an elastic information bottleneck (EIB) to interpolate between the IB and DIB regularizers, which guarantees a Pareto frontier within the IB framework. Additionally, simulations and real data experiments show that EIB has the ability to achieve better domain adaptation results than IB and DIB, which validates the correctness of our theories.