# Separable Computation of Information Measures

arXiv: https://arxiv.org/abs/2501.15301 · HTML: https://mlsanity.com/paper/2501.15301/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 25 Jan 2025.

## Builds on (most cited first, 5 of 5)

- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Approximating mutual information of high-dimensional variables using learned representations](https://mlsanity.com/paper/2409.02732/) (2024)
- [On Universal Features for High-Dimensional Learning and Inference](https://mlsanity.com/paper/1911.09105/) (2019)
- [Neural Feature Learning in Function Space](https://mlsanity.com/paper/2309.10140/) (2023)
- [Dependence Induced Representations](https://mlsanity.com/paper/2411.15328/) (2024)

## Abstract (arXiv)

We study a separable design for computing information measures, where the information measure is computed from learned feature representations instead of raw data. Under mild assumptions on the feature representations, we demonstrate that a class of information measures admit such separable computation, including mutual information, f-information, Wyner's common information, G'acs--K"orner common information, and Tishby's information bottleneck. Our development establishes several new connections between information measures and the statistical dependence structure. The characterizations also provide theoretical guarantees of practical designs for estimating information measures through representation learning.