# Flow of Information in Feed-Forward Deep Neural Networks

arXiv: https://arxiv.org/abs/1603.06220 · HTML: https://mlsanity.com/paper/1603.06220/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2016.

## Built on by (most cited first, 3 of 3)

- [Drill the Cork of Information Bottleneck by Inputting the Most Important Data](https://mlsanity.com/paper/2105.07181/) (2021)
- [Feedforward Ordering in Neural Connectomes via Feedback Arc Minimization](https://mlsanity.com/paper/2506.13799/) (2025)
- [Post-Training Neural Network Pruning using Graph Curvature](https://mlsanity.com/paper/2601.16366/) (2026)

## Abstract (arXiv)

Feed-forward deep neural networks have been used extensively in various machine learning applications. Developing a precise understanding of the underling behavior of neural networks is crucial for their efficient deployment. In this paper, we use an information theoretic approach to study the flow of information in a neural network and to determine how entropy of information changes between consecutive layers. Moreover, using the Information Bottleneck principle, we develop a constrained optimization problem that can be used in the training process of a deep neural network. Furthermore, we determine a lower bound for the level of data representation that can be achieved in a deep neural network with an acceptable level of distortion.