# On Correlation of Features Extracted by Deep Neural Networks

arXiv: https://arxiv.org/abs/1901.10900 · HTML: https://mlsanity.com/paper/1901.10900/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2019.

## Built on by (most cited first, 4 of 4)

- [Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective](https://mlsanity.com/paper/2206.07743/) (2022)
- [The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks](https://mlsanity.com/paper/2104.06153/) (2021)
- [UGC Quality Assessment: Exploring the Impact of Saliency in Deep Feature-Based Quality Assessment](https://mlsanity.com/paper/2308.06853/) (2023)
- [Diverse Imagenet Models Transfer Better](https://mlsanity.com/paper/2204.09134/) (2022)

## Abstract (arXiv)

Redundancy in deep neural network (DNN) models has always been one of their most intriguing and important properties. DNNs have been shown to overparameterize, or extract a lot of redundant features. In this work, we explore the impact of size (both width and depth), activation function, and weight initialization on the susceptibility of deep neural network models to extract redundant features. To estimate the number of redundant features in each layer, all the features of a given layer are hierarchically clustered according to their relative cosine distances in feature space and a set threshold. It is shown that both network size and activation function are the two most important components that foster the tendency of DNNs to extract redundant features. The concept is illustrated using deep multilayer perceptron and convolutional neural networks on MNIST digits recognition and CIFAR-10 dataset, respectively.