# Mechanisms of dimensionality reduction and decorrelation in deep neural networks

arXiv: https://arxiv.org/abs/1710.01467 · HTML: https://mlsanity.com/paper/1710.01467/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2017.

## Built on by (most cited first, 3 of 3)

- [Dense Hebbian neural networks: a replica symmetric picture of supervised learning](https://mlsanity.com/paper/2212.00606/) (2022)
- [Eight challenges in developing theory of intelligence](https://mlsanity.com/paper/2306.11232/) (2023)
- [Exploring Loss Landscapes through the Lens of Spin Glass Theory](https://mlsanity.com/paper/2407.20724/) (2024)

## Abstract (arXiv)

Deep neural networks are widely used in various domains. However, the nature of computations at each layer of the deep networks is far from being well understood. Increasing the interpretability of deep neural networks is thus important. Here, we construct a mean-field framework to understand how compact representations are developed across layers, not only in deterministic deep networks with random weights but also in generative deep networks where an unsupervised learning is carried out. Our theory shows that the deep computation implements a dimensionality reduction while maintaining a finite level of weak correlations between neurons for possible feature extraction. Mechanisms of dimensionality reduction and decorrelation are unified in the same framework. This work may pave the way for understanding how a sensory hierarchy works.