# Measuring multivariate redundant information with pointwise common change in surprisal

arXiv: https://arxiv.org/abs/1602.05063 · HTML: https://mlsanity.com/paper/1602.05063/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Feb 2016.

## Built on by (most cited first, 13 of 13)

- [A Measure of the Complexity of Neural Representations based on Partial Information Decomposition](https://mlsanity.com/paper/2209.10438/) (2022)
- [Learning Degradation-Independent Representations for Camera ISP Pipelines](https://mlsanity.com/paper/2307.00761/) (2023)
- [Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks](https://mlsanity.com/paper/2210.02996/) (2022)
- [A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions](https://mlsanity.com/paper/2306.02149/) (2023)
- [InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle](https://mlsanity.com/paper/2206.12292/) (2022)
- [Neural Dependency Coding inspired Multimodal Fusion](https://mlsanity.com/paper/2110.00385/) (2021)
- [Decomposing Interventional Causality into Synergistic, Redundant, and Unique Components](https://mlsanity.com/paper/2501.11447/) (2025)
- [Broadcast Channel Cooperative Gain: An Operational Interpretation of Partial Information Decomposition](https://mlsanity.com/paper/2502.10878/) (2025)
- [Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation](https://mlsanity.com/paper/2601.14274/) (2026)
- [Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models](https://mlsanity.com/paper/2605.08145/) (2026)
- [DIPHINE: Diffusion-based Phi-ID Neural Estimator](https://mlsanity.com/paper/2606.18997/) (2026)
- [Information Dynamics of Language Communication](https://mlsanity.com/paper/2606.30096/) (2026)
- [MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads](https://mlsanity.com/paper/2609.09206/) (2026)

## Abstract (arXiv)

The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S that is common to two or more predictor variables Xi. It can be thought of as quantifying overlapping information content or similarities in the representation of S between the Xi. We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We provide a game-theoretic operational definition of unique information, and use this to derive constraints which are used to obtain a maximum entropy distribution. Redundancy is then calculated from this maximum entropy distribution by counting only those local co-information terms which admit an unambiguous interpretation as redundant information. We show how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information into redundant, unique and synergistic contributions. We compare our new measure to existing approaches over a range of example systems, including continuous Gaussian variables. Matlab code for the measure is provided, including all considered examples.