mlsanityData to 7 Oct 2026

Approximation and Learning with Deep Convolutional Models: a Kernel Perspective

arXiv 2102.10032 ↗First posted 19 Feb 2021stat.ML cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

The empirical success of deep convolutional networks on tasks involving high-dimensional data such as images or audio suggests that they can efficiently approximate certain functions that are well-suited for such tasks. In this paper, we study this through the lens of kernel methods, by considering simple hierarchical kernels with two or three convolution and pooling layers, inspired by convolutional kernel networks. These achieve good empirical performance on standard vision datasets, while providing a precise description of their functional space that yields new insights on their inductive bias. We show that the RKHS consists of additive models of interaction terms between patches, and that its norm encourages spatial similarities between these terms through pooling layers. We then provide generalization bounds which illustrate how pooling and patches yield improved sample complexity guarantees when the target function presents such regularities.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 41 of its 71 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

41 of 71 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  7. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  9. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  12. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
29 more
  1. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  2. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  3. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  4. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  5. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  6. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  7. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  8. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  9. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  10. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  11. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  12. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  13. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  14. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
  15. 2020Neural Kernels Without Tangentscited by 56
  16. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  17. 2014Convolutional Kernel Networkscited by 47
  18. 2015A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extractioncited by 46
  19. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  20. 2019Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generatorscited by 40
  21. 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
  22. 2020Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?cited by 32
  23. 2016End-to-End Kernel Learning with Supervised Convolutional Kernel Networkscited by 28
  24. 2018How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?cited by 19
  25. 2020Computational Separation Between Convolutional and Fully-Connected Networkscited by 19
  26. 2016Convexified Convolutional Neural Networkscited by 9
  27. 2021The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methodscited by 8
  28. 2020Harmonic Decompositions of Convolutional Networkscited by 6
  29. 2018Localized Structured Predictioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 7; 2024: 7; 2025: 1; 2026: 2.

31 papers.

Sort:

By area:

  1. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  2. 2021Neural Contextual Bandits without Regretcited by 40
  3. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  4. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  5. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  6. 2023Theoretical Analysis of Inductive Biases in Deep Convolutional Networkscited by 21
  7. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  8. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  9. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  10. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 15
  11. 2023Mechanism of feature learning in convolutional neural networkscited by 15
  12. 2021On the Sample Complexity of Learning under Invariance and Geometric Stabilitycited by 13
19 more
  1. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  2. 2022A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momentacited by 11
  3. 2022Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditionscited by 9
  4. 2023Strong inductive biases provably prevent harmless interpolationcited by 9
  5. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  6. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  7. 2022On the Universal Approximation Property of Deep Fully Convolutional Neural Networkscited by 6
  8. 2022Transfer Learning with Kernel Methodscited by 5
  9. 2024Spectral complexity of deep neural networkscited by 3
  10. 2024Demystifying Spectral Bias on Real-World Datacited by 3
  11. 2022On the Shift Invariance of Max Pooling Feature Maps in Convolutional Neural Networkscited by 2
  12. 2023Sparse Gaussian Processes with Spherical Harmonic Features Revisitedcited by 2
  13. 2024SGD with memory: fundamental properties and stochastic accelerationcited by 2
  14. 2021A Johnson--Lindenstrauss Framework for Randomly Initialized CNNscited by 0
  15. 2023Kernels, Data & Physicscited by 0
  16. 2024Stability Analysis of Equivariant Convolutional Representations Through The Lens of Equivariant Multi-layered CKNscited by 0
  17. 2025Towards High-Fidelity Gaussian Splatting with Queried-Convolution Neural Networkscited by 0
  18. 2026Lipschitz bounds for integral kernelscited by 0
  19. 2026Theory Guided and Interpretable Neural Operator Design for Partial Differential Equation Learningcited by 0