# Surveying the space of descriptions of a composite system with machine learning

arXiv: https://arxiv.org/abs/2411.18579 · HTML: https://mlsanity.com/paper/2411.18579/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 61 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 16 of 16)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Mutual Information, Neural Networks and the Renormalization Group](https://mlsanity.com/paper/1704.06279/) (2017)
- [Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information](https://mlsanity.com/paper/1902.11239/) (2019)
- [The extent and drivers of gender imbalance in neuroscience reference lists](https://mlsanity.com/paper/2001.01002/) (2020)
- [A Measure of the Complexity of Neural Representations based on Partial Information Decomposition](https://mlsanity.com/paper/2209.10438/) (2022)
- [Information decomposition in complex systems via machine learning](https://mlsanity.com/paper/2307.04755/) (2023)
- [Partial entropy decomposition reveals higher-order structures in human brain activity](https://mlsanity.com/paper/2301.05307/) (2023)
- [Interpretability with full complexity by constraining feature information](https://mlsanity.com/paper/2211.17264/) (2022)
- [The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal](https://mlsanity.com/paper/1702.01591/) (2017)
- [Citation inequity and gendered citation practices in contemporary physics](https://mlsanity.com/paper/2112.09047/) (2021)
- [Machine-learning optimized measurements of chaotic dynamical systems via the information bottleneck](https://mlsanity.com/paper/2311.04896/) (2023)
- [Partial information decomposition: redundancy as information bottleneck](https://mlsanity.com/paper/2405.07665/) (2024)
- [Comparing the information content of probabilistic representation spaces](https://mlsanity.com/paper/2405.21042/) (2024)

## Abstract (arXiv)

Multivariate information theory provides a general and principled framework for understanding how the components of a complex system are connected. Existing analyses are coarse in nature -- built up from characterizations of discrete subsystems -- and can be computationally prohibitive. In this work, we propose to study the continuous space of possible descriptions of a composite system as a window into its organizational structure. A description consists of specific information conveyed about each of the components, and the space of possible descriptions is equivalent to the space of lossy compression schemes of the components. We introduce a machine learning framework to optimize descriptions that extremize key information theoretic quantities used to characterize organization, such as total correlation and O-information. Through case studies on spin systems, sudoku boards, and letter sequences from natural language, we identify extremal descriptions that reveal how system-wide variation emerges from individual components. By integrating machine learning into a fine-grained information theoretic analysis of composite random variables, our framework opens a new avenues for probing the structure of real-world complex systems.