# The Dual Information Bottleneck

arXiv: https://arxiv.org/abs/2006.04641 · HTML: https://mlsanity.com/paper/2006.04641/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2020.

## Built on by (most cited first, 14 of 14)

- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification](https://mlsanity.com/paper/2104.02862/) (2021)
- [Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors](https://mlsanity.com/paper/2205.10279/) (2022)
- [An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization](https://mlsanity.com/paper/2303.00633/) (2023)
- [MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective](https://mlsanity.com/paper/2204.04391/) (2022)
- [What Do We Maximize in Self-Supervised Learning?](https://mlsanity.com/paper/2207.10081/) (2022)
- [Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation](https://mlsanity.com/paper/2607.21973/) (2026)
- [Homological Neural Networks: A Sparse Architecture for Multivariate Complexity](https://mlsanity.com/paper/2306.15337/) (2023)
- [Robust Few-Shot Named Entity Recognition with Boundary Discrimination and Correlation Purification](https://mlsanity.com/paper/2312.07961/) (2023)
- [Variational Distillation for Multi-View Learning](https://mlsanity.com/paper/2206.09548/) (2022)
- [Dynamic Encoding and Decoding of Information for Split Learning in Mobile-Edge Computing: Leveraging Information Bottleneck Theory](https://mlsanity.com/paper/2309.02787/) (2023)
- [Label Smoothing is a Pragmatic Information Bottleneck](https://mlsanity.com/paper/2508.14077/) (2025)
- [SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis](https://mlsanity.com/paper/2609.05174/) (2026)

## Abstract (arXiv)

The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity. Here we present a new framework, the Dual Information Bottleneck (dualIB), which resolves some of the known drawbacks of the IB. We provide a theoretical analysis of the dualIB framework; (i) solving for the structure of its solutions (ii) unraveling its superiority in optimizing the mean prediction error exponent and (iii) demonstrating its ability to preserve exponential forms of the original distribution. To approach large scale problems, we present a novel variational formulation of the dualIB for Deep Neural Networks. In experiments on several data-sets, we compare it to a variational form of the IB. This exposes superior Information Plane properties of the dualIB and its potential in improvement of the error.