# Data efficiency, dimensionality reduction, and the generalized symmetric information bottleneck

arXiv: https://arxiv.org/abs/2309.05649 · HTML: https://mlsanity.com/paper/2309.05649/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 54 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 5 of 5)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [The deterministic information bottleneck](https://mlsanity.com/paper/1604.00268/) (2016)
- [Large-scale neural recordings call for new insights to link brain and behavior](https://mlsanity.com/paper/2103.14662/) (2021)
- [Multivariate Information Bottleneck](https://mlsanity.com/paper/1301.2270/) (2013)

## Built on by (most cited first, 5 of 5)

- [Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses](https://mlsanity.com/paper/2310.03311/) (2023)
- [Simultaneous Dimensionality Reduction: A Data Efficient Approach for Multimodal Representations Learning](https://mlsanity.com/paper/2310.04458/) (2023)
- [Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning](https://mlsanity.com/paper/2606.07770/) (2026)
- [Mutual information and task-relevant latent dimensionality](https://mlsanity.com/paper/2602.08105/) (2026)
- [Information bottleneck for learning the phase space of dynamics from high-dimensional experimental data](https://mlsanity.com/paper/2604.24662/) (2026)

## Abstract (arXiv)

The Symmetric Information Bottleneck (SIB), an extension of the more familiar Information Bottleneck, is a dimensionality reduction technique that simultaneously compresses two random variables to preserve information between their compressed versions. We introduce the Generalized Symmetric Information Bottleneck (GSIB), which explores different functional forms of the cost of such simultaneous reduction. We then explore the dataset size requirements of such simultaneous compression. We do this by deriving bounds and root-mean-squared estimates of statistical fluctuations of the involved loss functions. We show that, in typical situations, the simultaneous GSIB compression requires qualitatively less data to achieve the same errors compared to compressing variables one at a time. We suggest that this is an example of a more general principle that simultaneous compression is more data efficient than independent compression of each of the input variables.