# Disentangled Information Bottleneck

arXiv: https://arxiv.org/abs/2012.07372 · HTML: https://mlsanity.com/paper/2012.07372/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 24 of 24)

- [Graph Structure Learning with Variational Information Bottleneck](https://mlsanity.com/paper/2112.08903/) (2021)
- [PAC-Bayes Information Bottleneck](https://mlsanity.com/paper/2109.14509/) (2021)
- [Conditional Graph Information Bottleneck for Molecular Relational Learning](https://mlsanity.com/paper/2305.01520/) (2023)
- [An Information Criterion for Controlled Disentanglement of Multimodal Data](https://mlsanity.com/paper/2410.23996/) (2024)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering](https://mlsanity.com/paper/2209.06954/) (2022)
- [Recognizable Information Bottleneck](https://mlsanity.com/paper/2304.14618/) (2023)
- [Disentangled Text Representation Learning with Information-Theoretic Perspective for Adversarial Robustness](https://mlsanity.com/paper/2210.14957/) (2022)
- [Disentangled Noisy Correspondence Learning](https://mlsanity.com/paper/2408.05503/) (2024)
- [Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning](https://mlsanity.com/paper/2510.20644/) (2025)
- [Causal Information Bottleneck Boosts Adversarial Robustness of Deep Neural Network](https://mlsanity.com/paper/2210.14229/) (2022)
- [Data Encoding For Healthcare Data Democratisation and Information Leakage Prevention](https://mlsanity.com/paper/2305.03710/) (2023)
- [Disentangled Representation Learning with Transmitted Information Bottleneck](https://mlsanity.com/paper/2311.01686/) (2023)
- [Contrastive Learning and Adversarial Disentanglement for Privacy-Aware Task-Oriented Semantic Communication](https://mlsanity.com/paper/2410.22784/) (2024)
- [Disentangled Generation with Information Bottleneck for Few-Shot Learning](https://mlsanity.com/paper/2211.16185/) (2022)
- [Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training](https://mlsanity.com/paper/2402.01238/) (2024)
- [Refining the Information Bottleneck via Adversarial Information Separation](https://mlsanity.com/paper/2602.06549/) (2026)
- [Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck](https://mlsanity.com/paper/2303.14096/) (2023)
- [IENE: Identifying and Extrapolating the Node Environment for Out-of-Distribution Generalization on Graphs](https://mlsanity.com/paper/2406.00764/) (2024)
- [Interpretable High-order Knowledge Graph Neural Network for Predicting Synthetic Lethality in Human Cancers](https://mlsanity.com/paper/2503.06052/) (2025)
- [Label Smoothing is a Pragmatic Information Bottleneck](https://mlsanity.com/paper/2508.14077/) (2025)
- [XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision](https://mlsanity.com/paper/2601.21688/) (2026)
- [Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts](https://mlsanity.com/paper/2605.03348/) (2026)
- [Disentangling Representation using Attributes-based Gaussian Estimation for Medical Sound Diagnosis](https://mlsanity.com/paper/2608.29026/) (2026)

## Abstract (arXiv)

The information bottleneck (IB) method is a technique for extracting information that is relevant for predicting the target random variable from the source random variable, which is typically implemented by optimizing the IB Lagrangian that balances the compression and prediction terms. However, the IB Lagrangian is hard to optimize, and multiple trials for tuning values of Lagrangian multiplier are required. Moreover, we show that the prediction performance strictly decreases as the compression gets stronger during optimizing the IB Lagrangian. In this paper, we implement the IB method from the perspective of supervised disentangling. Specifically, we introduce Disentangled Information Bottleneck (DisenIB) that is consistent on compressing source maximally without target prediction performance loss (maximum compression). Theoretical and experimental results demonstrate that our method is consistent on maximum compression, and performs well in terms of generalization, robustness to adversarial attack, out-of-distribution detection, and supervised disentangling.