# Relevant sparse codes with variational information bottleneck

arXiv: https://arxiv.org/abs/1605.07332 · HTML: https://mlsanity.com/paper/1605.07332/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2016.

## Built on by (most cited first, 17 of 17)

- [Mutual Information State Intrinsic Control](https://mlsanity.com/paper/2103.08107/) (2021)
- [Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration](https://mlsanity.com/paper/2103.12300/) (2021)
- [Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2312.09539/) (2023)
- [Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality](https://mlsanity.com/paper/2208.03848/) (2022)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [Revisiting Multivariate Time Series Forecasting with Missing Values](https://mlsanity.com/paper/2509.23494/) (2025)
- [Adversarial Information Bottleneck](https://mlsanity.com/paper/2103.00381/) (2021)
- [Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes](https://mlsanity.com/paper/2306.12045/) (2023)
- [Online Deep Learning from Doubly-Streaming Data](https://mlsanity.com/paper/2204.11793/) (2022)
- [Generalized Information Bottleneck for Gaussian Variables](https://mlsanity.com/paper/2303.17762/) (2023)
- [β-CapsNet: Learning Disentangled Representation for CapsNet by Information Bottleneck](https://mlsanity.com/paper/2209.05239/) (2022)
- [A representation-learning game for classes of prediction tasks](https://mlsanity.com/paper/2403.06971/) (2024)
- [Structural Entropy Guided Probabilistic Coding](https://mlsanity.com/paper/2412.08841/) (2024)
- [MIST: Mutual Information Estimation Via Supervised Training](https://mlsanity.com/paper/2511.18945/) (2025)
- [Progressive Interpretation Synthesis: Interpreting Task Solving by Quantifying Previously Used and Unused Information](https://mlsanity.com/paper/2101.02879/) (2021)
- [CGIBNet: Bandwidth-constrained Communication with Graph Information Bottleneck in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2112.10374/) (2021)
- [Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck](https://mlsanity.com/paper/2303.14096/) (2023)

## Abstract (arXiv)

In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximizing a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelized versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y.