# Information decomposition in complex systems via machine learning

arXiv: https://arxiv.org/abs/2307.04755 · HTML: https://mlsanity.com/paper/2307.04755/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 77 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 18 of 18)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Nonlinear Information Bottleneck](https://mlsanity.com/paper/1705.02436/) (2017)
- [The Information Bottleneck Problem and Its Applications in Machine Learning](https://mlsanity.com/paper/2004.14941/) (2020)
- [Caveats for information bottleneck in deterministic scenarios](https://mlsanity.com/paper/1808.07593/) (2018)
- [Mutual Information, Neural Networks and the Renormalization Group](https://mlsanity.com/paper/1704.06279/) (2017)
- [Distributed Variational Representation Learning](https://mlsanity.com/paper/1807.04193/) (2018)
- [Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information](https://mlsanity.com/paper/1902.11239/) (2019)
- [On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views](https://mlsanity.com/paper/2002.00008/) (2020)
- [A Novel Approach to the Partial Information Decomposition](https://mlsanity.com/paper/1908.08642/) (2019)
- [Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic](https://mlsanity.com/paper/2008.09535/) (2020)
- [The extent and drivers of gender imbalance in neuroscience reference lists](https://mlsanity.com/paper/2001.01002/) (2020)
- [Quantitative Evaluation of Gender Bias in Astronomical Publications from Citation Counts](https://mlsanity.com/paper/1610.08984/) (2016)
- [Interpretability with full complexity by constraining feature information](https://mlsanity.com/paper/2211.17264/) (2022)
- [The Distributed Information Bottleneck reveals the explanatory structure of complex systems](https://mlsanity.com/paper/2204.07576/) (2022)

## Built on by (most cited first, 9 of 9)

- [Human-like object concept representations emerge naturally in multimodal large language models](https://mlsanity.com/paper/2407.01067/) (2024)
- [How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning](https://mlsanity.com/paper/2602.15580/) (2026)
- [Machine-learning optimized measurements of chaotic dynamical systems via the information bottleneck](https://mlsanity.com/paper/2311.04896/) (2023)
- [Which bits went where? Past and future transfer entropy decomposition with the information bottleneck](https://mlsanity.com/paper/2411.04992/) (2024)
- [From independent patches to coordinated attention: Controlling information flow in vision transformers](https://mlsanity.com/paper/2602.04784/) (2026)
- [Training deep physical neural networks with local physical information bottleneck](https://mlsanity.com/paper/2602.09569/) (2026)
- [Learning from the past: predicting critical transitions with machine learning trained on surrogates of historical data](https://mlsanity.com/paper/2410.09707/) (2024)
- [Surveying the space of descriptions of a composite system with machine learning](https://mlsanity.com/paper/2411.18579/) (2024)
- [Attribution via Distributional Paths for Information Revelation](https://mlsanity.com/paper/2606.03885/) (2026)

## Abstract (arXiv)

One of the fundamental steps toward understanding a complex system is identifying variation at the scale of the system's components that is most relevant to behavior on a macroscopic scale. Mutual information provides a natural means of linking variation across scales of a system due to its independence of functional relationship between observables. However, characterizing the manner in which information is distributed across a set of observables is computationally challenging and generally infeasible beyond a handful of measurements. Here we propose a practical and general methodology that uses machine learning to decompose the information contained in a set of measurements by jointly optimizing a lossy compression of each measurement. Guided by the distributed information bottleneck as a learning objective, the information decomposition identifies the variation in the measurements of the system state most relevant to specified macroscale behavior. We focus our analysis on two paradigmatic complex systems: a Boolean circuit and an amorphous material undergoing plastic deformation. In both examples, the large amount of entropy of the system state is decomposed, bit by bit, in terms of what is most related to macroscale behavior. The identification of meaningful variation in data, with the full generality brought by information theory, is made practical for studying the connection between micro- and macroscale structure in complex systems.