# Multivariate Dependence Beyond Shannon Information

arXiv: https://arxiv.org/abs/1609.01233 · HTML: https://mlsanity.com/paper/1609.01233/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2016.

## Built on by (most cited first, 4 of 4)

- [Improving Multimodal fusion via Mutual Dependency Maximisation](https://mlsanity.com/paper/2109.00922/) (2021)
- [Neural Dependency Coding inspired Multimodal Fusion](https://mlsanity.com/paper/2110.00385/) (2021)
- [Information-Theoretic Measures on Lattices for High-Order Interactions](https://mlsanity.com/paper/2408.07533/) (2024)
- [Applications of Entropy in Data Analysis and Machine Learning: A Review](https://mlsanity.com/paper/2503.02921/) (2025)

## Abstract (arXiv)

Accurately determining dependency structure is critical to discovering a system's causal organization. We recently showed that the transfer entropy fails in a key aspect of this---measuring information flow---due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that this is true of all such Shannon information measures when used to analyze multivariate dependencies. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful dependency structure within joint probability distributions. Therefore, such information measures are inadequate for discovering intrinsic causal relations. We close by demonstrating that such distributions exist across an arbitrary set of variables.