# Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information

arXiv: https://arxiv.org/abs/1902.11239 · HTML: https://mlsanity.com/paper/1902.11239/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2019.

## Built on by (most cited first, 30 of 30)

- [Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition](https://mlsanity.com/paper/2408.08944/) (2024)
- [Learning High-Order Relationships of Brain Regions](https://mlsanity.com/paper/2312.02203/) (2023)
- [Generative power of a protein language model trained on multiple sequence alignments](https://mlsanity.com/paper/2204.07110/) (2022)
- [Information decomposition in complex systems via machine learning](https://mlsanity.com/paper/2307.04755/) (2023)
- [The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity](https://mlsanity.com/paper/2608.13520/) (2026)
- [Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks](https://mlsanity.com/paper/2210.02996/) (2022)
- [MvHo-IB: Multi-View Higher-Order Information Bottleneck for Brain Disorder Diagnosis](https://mlsanity.com/paper/2507.02847/) (2025)
- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [Partial Information Decomposition for Data Interpretability and Feature Selection](https://mlsanity.com/paper/2405.19212/) (2024)
- [A Generalized Information Bottleneck Theory of Deep Learning](https://mlsanity.com/paper/2509.26327/) (2025)
- [FMMI: Flow Matching Mutual Information Estimation](https://mlsanity.com/paper/2511.08552/) (2025)
- [A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges](https://mlsanity.com/paper/2604.07650/) (2026)
- [Neural Dependency Coding inspired Multimodal Fusion](https://mlsanity.com/paper/2110.00385/) (2021)
- [Kernel-based Joint Independence Tests for Multivariate Stationary and Non-stationary Time Series](https://mlsanity.com/paper/2305.08529/) (2023)
- [Interaction Measures, Partition Lattices and Kernel Tests for High-Order Interactions](https://mlsanity.com/paper/2306.00904/) (2023)
- [SOmegaI: Score-based O-INFORMATION Estimation](https://mlsanity.com/paper/2402.05667/) (2024)
- [Partial information decomposition: redundancy as information bottleneck](https://mlsanity.com/paper/2405.07665/) (2024)
- [Dissipation alters modes of information encoding in small quantum reservoirs near criticality](https://mlsanity.com/paper/2412.18290/) (2024)
- [Towards Worst-Case Guarantees with Scale-Aware Interpretability](https://mlsanity.com/paper/2602.05184/) (2026)
- [Modeling Higher-Order Brain Interactions via a Multi-View Information Bottleneck Framework for fMRI-based Psychiatric Diagnosis](https://mlsanity.com/paper/2604.17713/) (2026)
- [Higher-order mutual information reveals synergistic sub-networks for multi-neuron importance](https://mlsanity.com/paper/2211.00416/) (2022)
- [Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Parity](https://mlsanity.com/paper/2601.22450/) (2026)
- [Surveying the space of descriptions of a composite system with machine learning](https://mlsanity.com/paper/2411.18579/) (2024)
- [Cooperative effects in feature importance of individual patterns: application to air pollutants and Alzheimer disease](https://mlsanity.com/paper/2508.00930/) (2025)
- [Information-theoretic signatures of causality in Bayesian networks and hypergraphs](https://mlsanity.com/paper/2512.20552/) (2025)
- [HyperPotter: Spell the Charm of High-Order Interactions in Audio Deepfake Detection](https://mlsanity.com/paper/2602.05670/) (2026)
- [Factorization Regret mediates compositional generalization in latent space](https://mlsanity.com/paper/2603.27134/) (2026)
- [Dynamic Vine Copulas: Detecting and Quantifying Time-Varying Higher-Order Interactions](https://mlsanity.com/paper/2605.03061/) (2026)
- [Capacity and Redundancy Trade-offs in Multi-Task Learning](https://mlsanity.com/paper/2607.16554/) (2026)
- [ALICE: In-context, Zero-shot, Mutual Information Estimation](https://mlsanity.com/paper/2609.34962/) (2026)

## Abstract (arXiv)

This article introduces a model-agnostic approach to study statistical synergy, a form of emergence in which patterns at large scales are not traceable from lower scales. Our framework leverages various multivariate extensions of Shannon's mutual information, and introduces the O-information as a metric capable of characterising synergy- and redundancy-dominated systems. We develop key analytical properties of the O-information, and study how it relates to other metrics of high-order interactions from the statistical mechanics and neuroscience literature. Finally, as a proof of concept, we use the proposed framework to explore the relevance of statistical synergy in Baroque music scores.