# Information Plane Analysis Visualization in Deep Learning via Transfer Entropy

arXiv: https://arxiv.org/abs/2404.01364 · HTML: https://mlsanity.com/paper/2404.01364/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 46 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 13 of 13)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle](https://mlsanity.com/paper/1802.09766/) (2018)
- [The Information Bottleneck Problem and Its Applications in Machine Learning](https://mlsanity.com/paper/2004.14941/) (2020)
- [Entropy and mutual information in models of deep neural networks](https://mlsanity.com/paper/1805.09785/) (2018)
- [On Information Plane Analyses of Neural Network Classifiers -- A Review](https://mlsanity.com/paper/2003.09671/) (2020)
- [Caveats for information bottleneck in deterministic scenarios](https://mlsanity.com/paper/1808.07593/) (2018)
- [Transfer entropy-based feedback improves performance in artificial neural networks](https://mlsanity.com/paper/1706.04265/) (2017)
- [On the Difference Between the Information Bottleneck and the Deep Information Bottleneck](https://mlsanity.com/paper/1912.13480/) (2019)
- [Learning in Feedforward Neural Networks Accelerated by Transfer Entropy](https://mlsanity.com/paper/2104.14616/) (2021)
- [Visual Knowledge Discovery with Artificial Intelligence: Challenges and Future Directions](https://mlsanity.com/paper/2205.01296/) (2022)
- [Deep learning, stochastic gradient descent and diffusion maps](https://mlsanity.com/paper/2204.01365/) (2022)

## Built on by (most cited first, 1 of 1)

- [Transfer Entropy in Graph Convolutional Neural Networks](https://mlsanity.com/paper/2406.06632/) (2024)

## Abstract (arXiv)

In a feedforward network, Transfer Entropy (TE) can be used to measure the influence that one layer has on another by quantifying the information transfer between them during training. According to the Information Bottleneck principle, a neural model's internal representation should compress the input data as much as possible while still retaining sufficient information about the output. Information Plane analysis is a visualization technique used to understand the trade-off between compression and information preservation in the context of the Information Bottleneck method by plotting the amount of information in the input data against the compressed representation. The claim that there is a causal link between information-theoretic compression and generalization, measured by mutual information, is plausible, but results from different studies are conflicting. In contrast to mutual information, TE can capture temporal relationships between variables. To explore such links, in our novel approach we use TE to quantify information transfer between neural layers and perform Information Plane analysis. We obtained encouraging experimental results, opening the possibility for further investigations.