mlsanityData to 7 Oct 2026

Mechanism of feature learning in convolutional neural networks

arXiv 2309.00570 ↗First posted 1 Sep 2023stat.ML cs.CV cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Understanding the mechanism of how convolutional neural networks learn features from image data is a fundamental problem in machine learning and computer vision. In this work, we identify such a mechanism. We posit the Convolutional Neural Feature Ansatz, which states that covariances of filters in any convolutional layer are proportional to the average gradient outer product (AGOP) taken with respect to patches of the input to that layer. We present extensive empirical evidence for our ansatz, including identifying high correlation between covariances of filters and patch-based AGOPs for convolutional layers in standard neural architectures, such as AlexNet, VGG, and ResNets pre-trained on ImageNet. We also provide supporting theoretical evidence. We then demonstrate the generality of our result by using the patch-based AGOP to enable deep feature learning in convolutional kernel machines. We refer to the resulting algorithm as (Deep) ConvRFM and show that our algorithm recovers similar features to deep convolutional networks including the notable emergence of edge detectors. Moreover, we find that Deep ConvRFM overcomes previously identified limitations of convolutional kernels, such as their inability to adapt to local signals in images and, as a result, leads to sizable performance improvement over fixed convolutional kernels.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

39 of 59 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  11. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  12. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
27 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  4. 2014How transferable are features in deep neural networks?cited by 1,216
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  7. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  8. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  9. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  10. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  11. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  12. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  13. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  14. 2019Cold Case: The Lost MNIST Digitscited by 61
  15. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  16. 2020Neural Kernels Without Tangentscited by 56
  17. 2017Gabor Convolutional Networkscited by 55
  18. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  19. 2022Limitations of the NTK for Understanding Generalization in Deep Learningcited by 34
  20. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  21. 2016Lets keep it simple, Using simple architectures to outperform deeper and more complex architecturescited by 24
  22. 2021Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networkscited by 17
  23. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  24. 2022Understanding the Covariance Structure of Convolutional Filterscited by 16
  25. 2023Kernel Regression with Infinite-Width Neural Networks on Millions of Examplescited by 9
  26. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
  27. 2020The Expected Jacobian Outerproduct: Theory and Empiricscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 1; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  2. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  3. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  4. 2024Residual Connections Harm Generative Representation Learningcited by 10
  5. 2024Formation of Representations in Neural Networkscited by 10
  6. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 9
  7. 2023On the Nystrom Approximation for Preconditioning in Kernel Machinescited by 2
  8. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  9. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1
  10. 2026Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernelscited by 1
  11. 2026AGOP as Explanation: From Feature Learning to Per-Sample Attribution in Image Classifierscited by 1
  12. 2026Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligencecited by 1
3 more
  1. 2024Impact of Label Noise on Learning Complex Featurescited by 0
  2. 2024Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theoriescited by 0
  3. 2026Local EGOP for Continuous Index Learningcited by 0