# Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs

arXiv: https://arxiv.org/abs/2110.11400 · HTML: https://mlsanity.com/paper/2110.11400/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality](https://mlsanity.com/paper/1801.02613/) (2018)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units](https://mlsanity.com/paper/1603.05201/) (2016)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Regularizing CNNs with Locally Constrained Decorrelations](https://mlsanity.com/paper/1611.01967/) (2016)
- [Dimensionality compression and expansion in Deep Neural Networks](https://mlsanity.com/paper/1906.00443/) (2019)
- [Deep geometric knowledge distillation with graphs](https://mlsanity.com/paper/1911.03080/) (2019)
- [Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation](https://mlsanity.com/paper/2107.12972/) (2021)
- [Laplacian Networks: Bounding Indicator Function Smoothness for Neural Network Robustness](https://mlsanity.com/paper/1805.10133/) (2018)
- [Representing Deep Neural Networks Latent Space Geometries with Graphs](https://mlsanity.com/paper/2011.07343/) (2020)
- [DeepNNK: Explaining deep models and their generalization using polytope interpolation](https://mlsanity.com/paper/2007.10505/) (2020)
- [Efficient graph construction for image representation](https://mlsanity.com/paper/2002.06662/) (2020)

## Built on by (most cited first, 3 of 3)

- [The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning](https://mlsanity.com/paper/2209.08622/) (2022)
- [GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning](https://mlsanity.com/paper/2410.15266/) (2024)
- [Study of Manifold Geometry using Multiscale Non-Negative Kernel Graphs](https://mlsanity.com/paper/2210.17475/) (2022)

## Abstract (arXiv)

Feature spaces in the deep layers of convolutional neural networks (CNNs) are often very high-dimensional and difficult to interpret. However, convolutional layers consist of multiple channels that are activated by different types of inputs, which suggests that more insights may be gained by studying the channels and how they relate to each other. In this paper, we first analyze theoretically channel-wise non-negative kernel (CW-NNK) regression graphs, which allow us to quantify the overlap between channels and, indirectly, the intrinsic dimension of the data representation manifold. We find that redundancy between channels is significant and varies with the layer depth and the level of regularization during training. Additionally, we observe that there is a correlation between channel overlap in the last convolutional layer and generalization performance. Our experimental results demonstrate that these techniques can lead to a better understanding of deep representations.