mlsanityData to 7 Oct 2026

Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning

arXiv 2402.08010 ↗First posted 12 Feb 2024cs.LG cs.AI stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

We describe the emergence of a Convolution Bottleneck (CBN) structure in CNNs, where the network uses its first few layers to transform the input representation into a representation that is supported only along a few frequencies and channels, before using the last few layers to map back to the outputs. We define the CBN rank, which describes the number and type of frequencies that are kept inside the bottleneck, and partially prove that the parameter norm required to represent a function f scales as depth times the CBN rank f. We also show that the parameter norm depends at next order on the regularity of f. We show that any network with almost optimal parameter norm will exhibit a CBN structure in both the weights and - under the assumption that the network is stable under large learning rate - the activations, which motivates the common practice of down-sampling; and we verify that the CBN results still hold with down-sampling. Finally we use the CBN structure to interpret the functions learned by CNNs on a number of tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

22 of 32 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  3. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  6. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  7. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  8. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
  9. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  10. 2018Universal approximations of invariant maps by neural networkscited by 127
  11. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  12. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
10 more
  1. 2021Learning with invariances in random features and kernel modelscited by 89
  2. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  3. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  4. 2020Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?cited by 32
  5. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  6. 2023Theoretical Analysis of Inductive Biases in Deep Convolutional Networkscited by 21
  7. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  8. 2021Relative stability toward diffeomorphisms indicates performance in deep netscited by 12
  9. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  10. 2022Understanding robustness and generalization of artificial neural networks through Fourier maskscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 6; 2025: 3.

9 papers.

  1. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  2. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  3. 2024How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learningcited by 6
  4. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  5. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  6. 2024Geometric Inductive Biases of Deep Networks: The Role of Data and Architecturecited by 2
  7. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1
  8. 2025Fast Fourier Correlation is a Highly Efficient and Accurate Feature Attribution Algorithm from the Perspective of Control Theory and Game Theorycited by 1
  9. 2024Comprehensive and Reliable Feature Attribution for Diverse Modalities and Models via Frequency-Domain Insightscited by 0