mlsanityData to 14 Oct 2026

What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective

arXiv 2206.02454 ↗First posted 6 Jun 2022cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

It has previously been reported that the representation that is learned in the first layer of deep Convolutional Neural Networks (CNNs) is highly consistent across initializations and architectures. In this work, we quantify this consistency by considering the first layer as a filter bank and measuring its energy distribution. We find that the energy distribution is very different from that of the initial weights and is remarkably consistent across random initializations, datasets, architectures and even when the CNNs are trained with random labels. In order to explain this consistency, we derive an analytical formula for the energy profile of linear CNNs and show that this profile is mostly dictated by the second order statistics of image patches in the training set and it will approach a whitening transformation when the number of iterations goes to infinity. Finally, we show that this formula for linear CNNs also gives an excellent fit for the energy profiles learned by commonly used nonlinear CNNs such as ResNet and VGG, and that the first layer of these CNNs indeed perform approximate whitening of their inputs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

32 of 48 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  4. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  7. 2018On the Spectral Bias of Neural Networkscited by 1,341
  8. 2014How transferable are features in deep neural networks?cited by 1,225
  9. 2017A Closer Look at Memorization in Deep Networkscited by 942
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  12. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
20 more
  1. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  2. 2014Understanding image representations by measuring their equivariance and equivalencecited by 243
  3. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  4. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  5. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  6. 2019Switchable Whitening for Deep Representation Learningcited by 106
  7. 2018Decorrelated Batch Normalizationcited by 83
  8. 2021Similarity and Matching of Neural Network Representationscited by 78
  9. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  10. 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
  11. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 59
  12. 2017Gabor Convolutional Networkscited by 55
  13. 2015Natural Neural Networkscited by 44
  14. 2020Hierarchical nucleation in deep neural networkscited by 31
  15. 2021Grounding Representation Similarity with Statistical Testingcited by 29
  16. 2019GaborNet: Gabor filters with learnable parameters in deep convolutional neural networkscited by 17
  17. 2019Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networkscited by 11
  18. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  19. 2021Stochastic Whitening Batch Normalizationcited by 9
  20. 2017Gabor Filter Assisted Energy Efficient Fast Learning Convolutional Neural Networkscited by 8

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024How Do the Architecture and Optimizer Affect Representation Learning? On the Training Dynamics of Representations in Deep Neural Networkscited by 2