# The Convex Information Bottleneck Lagrangian

arXiv: https://arxiv.org/abs/1911.11000 · HTML: https://mlsanity.com/paper/1911.11000/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2019.

## Built on by (most cited first, 16 of 16)

- [An Information Criterion for Controlled Disentanglement of Multimodal Data](https://mlsanity.com/paper/2410.23996/) (2024)
- [The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning](https://mlsanity.com/paper/2307.10907/) (2023)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Recognizable Information Bottleneck](https://mlsanity.com/paper/2304.14618/) (2023)
- [Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning](https://mlsanity.com/paper/2510.20644/) (2025)
- [Disentangled Representation Learning with Transmitted Information Bottleneck](https://mlsanity.com/paper/2311.01686/) (2023)
- [Partial information decomposition: redundancy as information bottleneck](https://mlsanity.com/paper/2405.07665/) (2024)
- [Structured IB: Improving Information Bottleneck with Structured Feature Learning](https://mlsanity.com/paper/2412.08222/) (2024)
- [REMEDI: Corrective Transformations for Improved Neural Entropy Estimation](https://mlsanity.com/paper/2402.05718/) (2024)
- [Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors](https://mlsanity.com/paper/2502.15540/) (2025)
- [Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck](https://mlsanity.com/paper/2203.02592/) (2022)
- [Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training](https://mlsanity.com/paper/2402.01238/) (2024)
- [Multiview Representation Learning via Distributed Joint Latent Space Structuring](https://mlsanity.com/paper/2504.18455/) (2025)
- [Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck](https://mlsanity.com/paper/2303.14096/) (2023)
- [Stable and Convexified Information Bottleneck Optimization via Symbolic Continuation and Entropy-Regularized Trajectories](https://mlsanity.com/paper/2505.09239/) (2025)

## Abstract (arXiv)

The information bottleneck (IB) problem tackles the issue of obtaining relevant compressed representations T of some random variable X for the task of predicting Y. It is defined as a constrained optimization problem which maximizes the information the representation has about the task, I(T;Y), while ensuring that a certain level of compression r is achieved (i.e.,  I(X;T) ≤ r). For practical reasons, the problem is usually solved by maximizing the IB Lagrangian (i.e., LIB(T;β) = I(T;Y) - β I(X;T)) for many values of β ∈ [0,1]. Then, the curve of maximal I(T;Y) for a given I(X;T) is drawn and a representation with the desired predictability and compression is selected. It is known when Y is a deterministic function of X, the IB curve cannot be explored and another Lagrangian has been proposed to tackle this problem: the squared IB Lagrangian: Lsq-IB(T;βsq)=I(T;Y)-βsqI(X;T)2. In this paper, we (i) present a general family of Lagrangians which allow for the exploration of the IB curve in all scenarios; (ii) provide the exact one-to-one mapping between the Lagrange multiplier and the desired compression rate r for known IB curve shapes; and (iii) show we can approximately obtain a specific compression level with the convex IB Lagrangian for both known and unknown IB curve shapes. This eliminates the burden of solving the optimization problem for many values of the Lagrange multiplier. That is, we prove that we can solve the original constrained problem with a single optimization.