# A Local Geometric Interpretation of Feature Extraction in Deep Feedforward Neural Networks

arXiv: https://arxiv.org/abs/2202.04632 · HTML: https://mlsanity.com/paper/2202.04632/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 11 of 11)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [Geometric Understanding of Deep Learning](https://mlsanity.com/paper/1805.10451/) (2018)
- [A Minimax Approach to Supervised Learning](https://mlsanity.com/paper/1606.02206/) (2016)
- [On Universal Features for High-Dimensional Learning and Inference](https://mlsanity.com/paper/1911.09105/) (2019)
- [An Information Theoretic Interpretation to Deep Neural Networks](https://mlsanity.com/paper/1905.06600/) (2019)

## Built on by (most cited first, 2 of 2)

- [Timely Communications for Remote Inference](https://mlsanity.com/paper/2404.16281/) (2024)
- [How Does Data Freshness Affect Real-time Supervised Learning?](https://mlsanity.com/paper/2208.06948/) (2022)

## Abstract (arXiv)

In this paper, we present a local geometric analysis to interpret how deep feedforward neural networks extract low-dimensional features from high-dimensional data. Our study shows that, in a local geometric region, the optimal weight in one layer of the neural network and the optimal feature generated by the previous layer comprise a low-rank approximation of a matrix that is determined by the Bayes action of this layer. This result holds (i) for analyzing both the output layer and the hidden layers of the neural network, and (ii) for neuron activation functions with non-vanishing gradients. We use two supervised learning problems to illustrate our results: neural network based maximum likelihood classification (i.e., softmax regression) and neural network based minimum mean square estimation. Experimental validation of these theoretical results will be conducted in our future work.