mlsanityData to 14 Oct 2026

Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs

arXiv 2110.11400 ↗First posted 18 Oct 2021cs.LG cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Feature spaces in the deep layers of convolutional neural networks (CNNs) are often very high-dimensional and difficult to interpret. However, convolutional layers consist of multiple channels that are activated by different types of inputs, which suggests that more insights may be gained by studying the channels and how they relate to each other. In this paper, we first analyze theoretically channel-wise non-negative kernel (CW-NNK) regression graphs, which allow us to quantify the overlap between channels and, indirectly, the intrinsic dimension of the data representation manifold. We find that redundancy between channels is significant and varies with the layer depth and the level of regularization during training. Additionally, we observe that there is a correlation between channel overlap in the last convolutional layer and generalization performance. Our experimental results demonstrate that these techniques can lead to a better understanding of deep representations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

18 of 31 references.

References · 18

  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  2. 2020Designing Network Design Spacescited by 960
  3. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  4. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  5. 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 282
  6. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  7. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  8. 2019Orthogonal Convolutional Neural Networkscited by 123
  9. 2016Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Unitscited by 86
  10. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  11. 2016Regularizing CNNs with Locally Constrained Decorrelationscited by 41
  12. 2019Dimensionality compression and expansion in Deep Neural Networkscited by 38
6 more
  1. 2019Deep geometric knowledge distillation with graphscited by 21
  2. 2021Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolationcited by 11
  3. 2018Laplacian Networks: Bounding Indicator Function Smoothness for Neural Network Robustnesscited by 7
  4. 2020Representing Deep Neural Networks Latent Space Geometries with Graphscited by 7
  5. 2020DeepNNK: Explaining deep models and their generalization using polytope interpolationcited by 5
  6. 2020Efficient graph construction for image representationcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1.

3 papers.

  1. 2022The Geometry of Self-supervised Learning Models and its Impact on Transfer Learningcited by 7
  2. 2024GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learningcited by 4
  3. 2022Study of Manifold Geometry using Multiscale Non-Negative Kernel Graphscited by 1