# Learning Optimal Representations with the Decodable Information Bottleneck

arXiv: https://arxiv.org/abs/2009.12789 · HTML: https://mlsanity.com/paper/2009.12789/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2020.

## Built on by (most cited first, 30 of 33)

- [Graph Structure Learning with Variational Information Bottleneck](https://mlsanity.com/paper/2112.08903/) (2021)
- [Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2110.06530/) (2021)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Optimal Representations for Covariate Shift](https://mlsanity.com/paper/2201.00057/) (2021)
- [Compressive Visual Representations](https://mlsanity.com/paper/2109.12909/) (2021)
- [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://mlsanity.com/paper/2209.06235/) (2022)
- [VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution](https://mlsanity.com/paper/2304.01434/) (2023)
- [Gacs-Korner Common Information Variational Autoencoder](https://mlsanity.com/paper/2205.12239/) (2022)
- [Minimum Description Length and Generalization Guarantees for Representation Learning](https://mlsanity.com/paper/2402.03254/) (2024)
- [Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy](https://mlsanity.com/paper/2302.10888/) (2023)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [CRAFT: Adapting VLA Models to Contact-rich Manipulation via Force-aware Curriculum Fine-tuning](https://mlsanity.com/paper/2602.12532/) (2026)
- [Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering](https://mlsanity.com/paper/2209.06954/) (2022)
- [Perturbation Theory for the Information Bottleneck](https://mlsanity.com/paper/2105.13977/) (2021)
- [Improving VAE-based Representation Learning](https://mlsanity.com/paper/2205.14539/) (2022)
- [Evaluating Self-Supervised Learning via Risk Decomposition](https://mlsanity.com/paper/2302.03068/) (2023)
- [MLEM: Generative and Contrastive Learning as Distinct Modalities for Event Sequences](https://mlsanity.com/paper/2401.15935/) (2024)
- [Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey](https://mlsanity.com/paper/2204.08226/) (2022)
- [Recognizable Information Bottleneck](https://mlsanity.com/paper/2304.14618/) (2023)
- [Domain Generalization without Excess Empirical Risk](https://mlsanity.com/paper/2308.15856/) (2023)
- [Multi-Agent Reinforcement Learning with Communication-Constrained Priors](https://mlsanity.com/paper/2512.03528/) (2025)
- [SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data](https://mlsanity.com/paper/2210.02989/) (2022)
- [Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors](https://mlsanity.com/paper/2502.15540/) (2025)
- [Elastic Information Bottleneck](https://mlsanity.com/paper/2311.03955/) (2023)
- [A representation-learning game for classes of prediction tasks](https://mlsanity.com/paper/2403.06971/) (2024)
- [Multiview Representation Learning via Distributed Joint Latent Space Structuring](https://mlsanity.com/paper/2504.18455/) (2025)
- [Data coarse graining can improve model performance](https://mlsanity.com/paper/2509.14498/) (2025)
- [Attribute Graphs Underlying Molecular Generative Models: Path to Learning with Limited Data](https://mlsanity.com/paper/2207.07174/) (2022)
- [DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models](https://mlsanity.com/paper/2604.15016/) (2026)
- [The Bioelectrical Information Theory: Investigating the theoretical compression limit of bioelectrical signals under artificial intelligence](https://mlsanity.com/paper/2606.09922/) (2026)

## Abstract (arXiv)

We address the question of characterizing and finding optimal representations for supervised learning. Traditionally, this question has been tackled using the Information Bottleneck, which compresses the inputs while retaining information about the targets, in a decoder-agnostic fashion. In machine learning, however, our goal is not compression but rather generalization, which is intimately linked to the predictive family or decoder of interest (e.g. linear classifier). We propose the Decodable Information Bottleneck (DIB) that considers information retention and compression from the perspective of the desired predictive family. As a result, DIB gives rise to representations that are optimal in terms of expected test performance and can be estimated with guarantees. Empirically, we show that the framework can be used to enforce a small generalization gap on downstream classifiers and to predict the generalization ability of neural networks.