# An Information Theoretic Interpretation to Deep Neural Networks

arXiv: https://arxiv.org/abs/1905.06600 · HTML: https://mlsanity.com/paper/1905.06600/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2019.

## Built on by (most cited first, 19 of 19)

- [Understanding polysemanticity in neural networks through coding theory](https://mlsanity.com/paper/2401.17975/) (2024)
- [Operator SVD with Neural Networks via Nested Low-Rank Approximation](https://mlsanity.com/paper/2402.03655/) (2024)
- [Activation Learning by Local Competitions](https://mlsanity.com/paper/2209.13400/) (2022)
- [H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer](https://mlsanity.com/paper/2312.12489/) (2023)
- [Jet Expansions of Residual Computation](https://mlsanity.com/paper/2410.06024/) (2024)
- [Demystifying Spectral Feature Learning for Instrumental Variable Regression](https://mlsanity.com/paper/2506.10899/) (2025)
- [Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation](https://mlsanity.com/paper/2302.02521/) (2023)
- [A Local Geometric Interpretation of Feature Extraction in Deep Feedforward Neural Networks](https://mlsanity.com/paper/2202.04632/) (2022)
- [Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning](https://mlsanity.com/paper/2305.14397/) (2023)
- [Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings](https://mlsanity.com/paper/2502.11609/) (2025)
- [Information Science Principles of Machine Learning: A Causal Chain Meta-Framework Based on Formalized Information Mapping](https://mlsanity.com/paper/2505.13182/) (2025)
- [Information Geometry of Evolution of Neural Network Parameters While Training](https://mlsanity.com/paper/2406.05295/) (2024)
- [Slope and generalization properties of neural networks](https://mlsanity.com/paper/2107.01473/) (2021)
- [Kernel Subspace and Feature Extraction](https://mlsanity.com/paper/2301.01410/) (2023)
- [Protecting Activity Sensing Data Privacy Using Hierarchical Information Dissociation](https://mlsanity.com/paper/2409.03796/) (2024)
- [Learning Optimal Prompt Ensemble for Multi-source Visual Prompt Transfer](https://mlsanity.com/paper/2504.12311/) (2025)
- [The Effective Depth Paradox: Topology and Trainability in Deep CNNs](https://mlsanity.com/paper/2602.13298/) (2026)
- [A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis](https://mlsanity.com/paper/2603.11428/) (2026)
- [Universal Feature Selection with Noisy Observations and Weak Symmetry Conditions](https://mlsanity.com/paper/2605.09396/) (2026)

## Abstract (arXiv)

It is commonly believed that the hidden layers of deep neural networks (DNNs) attempt to extract informative features for learning tasks. In this paper, we formalize this intuition by showing that the features extracted by DNN coincide with the result of an optimization problem, which we call the `universal feature selection' problem, in a local analysis regime. We interpret the weights training in DNN as the projection of feature functions between feature spaces, specified by the network structure. Our formulation has direct operational meaning in terms of the performance for inference tasks, and gives interpretations to the internal computation results of DNNs. Results of numerical experiments are provided to support the analysis.