# SOmegaI: Score-based O-INFORMATION Estimation

arXiv: https://arxiv.org/abs/2402.05667 · HTML: https://mlsanity.com/paper/2402.05667/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [Interpretable Diffusion via Information Decomposition](https://mlsanity.com/paper/2310.07972/) (2023)
- [How Much is Enough? A Study on Diffusion Times in Score-based Generative Models](https://mlsanity.com/paper/2206.05173/) (2022)
- [MINDE: Mutual Information Neural Diffusion Estimation](https://mlsanity.com/paper/2310.09031/) (2023)
- [Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information](https://mlsanity.com/paper/1902.11239/) (2019)
- [Gaussian Partial Information Decomposition: Bias Correction and Application to High-dimensional Data](https://mlsanity.com/paper/2307.10515/) (2023)
- [Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems](https://mlsanity.com/paper/1411.2832/) (2014)
- [Multi-modal Latent Diffusion](https://mlsanity.com/paper/2306.04445/) (2023)
- [Multivariate Information Theory Uncovers Synergistic Subsystems of the Human Cerebral Cortex](https://mlsanity.com/paper/2206.06477/) (2022)
- [Partial entropy decomposition reveals higher-order structures in human brain activity](https://mlsanity.com/paper/2301.05307/) (2023)
- [Quantifying dynamical high-order interdependencies from the O-information: an application to neural spiking dynamics](https://mlsanity.com/paper/2007.16018/) (2020)

## Built on by (most cited first, 3 of 3)

- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [FMMI: Flow Matching Mutual Information Estimation](https://mlsanity.com/paper/2511.08552/) (2025)
- [Improved Sampling Schedules for Discrete Diffusion Models](https://mlsanity.com/paper/2602.06849/) (2026)

## Abstract (arXiv)

The analysis of scientific data and complex multivariate systems requires information quantities that capture relationships among multiple random variables. Recently, new information-theoretic measures have been developed to overcome the shortcomings of classical ones, such as mutual information, that are restricted to considering pairwise interactions. Among them, the concept of information synergy and redundancy is crucial for understanding the high-order dependencies between variables. One of the most prominent and versatile measures based on this concept is O-information, which provides a clear and scalable way to quantify the synergy-redundancy balance in multivariate systems. However, its practical application is limited to simplified cases. In this work, we introduce SOmegaI, which allows for the first time to compute O-information without restrictive assumptions about the system. Our experiments validate our approach on synthetic data, and demonstrate the effectiveness of SOmegaI in the context of a real-world use case.