mlsanityData to 7 Oct 2026

Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequencies

arXiv 2303.02034 ↗First posted 3 Mar 2023cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We here present a stepping stone towards a deeper understanding of convolutional neural networks (CNNs) in the form of a theory of learning in linear CNNs. Through analyzing the gradient descent equations, we discover that the evolution of the network during training is determined by the interplay between the dataset structure and the convolutional network structure. We show that linear CNNs discover the statistical structure of the dataset with non-linear, ordered, stage-like transitions, and that the speed of discovery changes depending on the relationship between the dataset and the convolutional network structure. Moreover, we find that this interplay lies at the heart of what we call the ``dominant frequency bias'', where linear CNNs arrive at these discoveries using only the dominant frequencies of the different structural parts present in the dataset. We furthermore provide experiments that show how our theory relates to deep, non-linear CNNs used in practice. Our findings shed new light on the inner working of CNNs, and can help explain their shortcut learning and their tendency to rely on texture instead of shape.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

28 of 51 references.

References · 28

  1. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  3. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  4. 2018On the Spectral Bias of Neural Networkscited by 1,321
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  12. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
16 more
  1. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  2. 2018Training behavior of deep neural network in frequency domaincited by 174
  3. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  4. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  5. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  6. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  7. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  8. 2018The Singular Values of Convolutional Layerscited by 95
  9. 2017Measuring the tendency of CNNs to Learn Surface Statistical Regularitiescited by 87
  10. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  11. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
  12. 2018On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functionscited by 31
  13. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  14. 2018Learning Dynamics of Linear Denoising Autoencoderscited by 16
  15. 2021The Dynamics of Gradient Descent for Overparametrized Neural Networkscited by 11
  16. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025FreqDebias: Towards Generalizable Deepfake Detection via Consistency-Driven Frequency Debiasingcited by 14
  2. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  3. 2024Towards Combating Frequency Simplicity-biased Learning for Domain Generalizationcited by 6
  4. 2024Spectral Norm of Convolutional Layers with Circular and Zero Paddingscited by 4
  5. 2026It's Not a Lottery, It's a Race: Understanding How Gradient Descent Adapts the Network's Capacity to the Taskcited by 1
  6. 2025D-GAP: Improving Out-of-Domain Robustness via Dataset-Agnostic and Gradient-Guided Augmentation in Frequency and Pixel Spacescited by 0
  7. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0