mlsanityData to 7 Oct 2026

What Can Be Learnt With Wide Convolutional Neural Networks?

arXiv 2208.01003 ↗First posted 1 Aug 2022stat.ML cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional functions remains a fundamental challenge. A popular belief is that these models harness the local and hierarchical structure of natural data such as images. Yet, we lack a quantitative understanding of how such structure affects performance, e.g., the rate of decay of the generalisation error with the number of training samples. In this paper, we study infinitely-wide deep CNNs in the kernel regime. First, we show that the spectrum of the corresponding kernel inherits the hierarchical structure of the network, and we characterise its asymptotics. Then, we use this result together with generalisation bounds to prove that deep CNNs adapt to the spatial scale of the target function. In particular, we find that if the target function depends on low-dimensional subsets of adjacent input variables, then the decay of the error is controlled by the effective dimensionality of these subsets. Conversely, if the target function depends on the full set of input variables, then the error decay is controlled by the input dimension. We conclude by computing the generalisation error of a deep CNN trained on the output of another deep CNN with randomly-initialised parameters. Interestingly, we find that, despite their hierarchical structure, the functions generated by infinitely-wide deep CNNs are too rich to be efficiently learnable in high dimension.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 39 of its 72 references that are in the map. First posted on arXiv 1 Aug 2022.

Builds on

39 of 72 references.

References · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  8. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  9. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  12. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
27 more
  1. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  2. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 188
  3. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  4. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  6. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  7. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  8. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  9. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  10. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  11. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  12. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  13. 2020Implicit Regularization of Random Feature Modelscited by 54
  14. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  15. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  16. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  17. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  18. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  19. 2020Computational Separation Between Convolutional and Fully-Connected Networkscited by 19
  20. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  21. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 15
  22. 2021Relative stability toward diffeomorphisms indicates performance in deep netscited by 12
  23. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12
  24. 2020On the rate of convergence of image classifiers based on convolutional neural networkscited by 7
  25. 2022On the inability of Gaussian process regression to optimally learn compositional functionscited by 7
  26. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  27. 2018Building Bayesian Neural Networks with Blocks: On Structure, Interpretability and Uncertaintycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 4; 2026: 1.

16 papers.

Sort:

By area:

  1. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  2. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  3. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 38
  4. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 24
  5. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  6. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  7. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  8. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  9. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  10. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  11. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  12. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
4 more
  1. 2024Spectral complexity of deep neural networkscited by 3
  2. 2024Demystifying Spectral Bias on Real-World Datacited by 3
  3. 2026A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernelcited by 1
  4. 2025Asymptotic convexity of wide and shallow neural networkscited by 0