mlsanityData to 14 Oct 2026

Locality defeats the curse of dimensionality in convolutional teacher-student scenarios

arXiv 2106.08619 ↗First posted 16 Jun 2021stat.ML cond-mat.dis-nn cs.LGcited by 26

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Convolutional neural networks perform a local and translationally-invariant treatment of the data: quantifying which of these two aspects is central to their success remains a challenge. We study this problem within a teacher-student framework for kernel regression, using `convolutional' kernels inspired by the neural tangent kernel of simple convolutional architectures of given filter size. Using heuristic methods from physics, we find in the ridgeless case that locality is key in determining the learning curve exponent β (that relates the test error εt~ P-β to the size of the training set P), whereas translational invariance is not. In particular, if the filter size of the teacher t is smaller than that of the student s, β is a function of s only and does not depend on the input dimension. We confirm our predictions on β empirically. We conclude by proving, using a natural universality assumption, that performing kernel regression with a ridge that decreases with the size of the training set leads to similar learning curve exponents to those we obtain in the ridgeless case.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 27 of its 47 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

27 of 47 references.

References · 27

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  5. 2017Deep Neural Networks as Gaussian Processescited by 622
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  9. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  10. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  11. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  12. 2018Gaussian Processes and Kernel Methods: A Review on Connections and Equivalencescited by 192
15 more
  1. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  2. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  3. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  4. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  5. 2021Learning with invariances in random features and kernel modelscited by 89
  6. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  7. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  8. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 67
  9. 2020Implicit Regularization of Random Feature Modelscited by 54
  10. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  11. 2020Towards Learning Convolutions from Scratchcited by 46
  12. 2016End-to-End Kernel Learning with Supervised Convolutional Kernel Networkscited by 28
  13. 2019Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigmcited by 20
  14. 2020Computational Separation Between Convolutional and Fully-Connected Networkscited by 20
  15. 2018Building Bayesian Neural Networks with Blocks: On Structure, Interpretability and Uncertaintycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 8; 2024: 3; 2025: 2; 2026: 1.

26 papers.

Sort:

By area:

  1. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  2. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datacited by 93
  3. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 71
  4. 2023The SSL Interplay: Augmentations, Inductive Bias, and Generalizationcited by 40
  5. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  6. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  7. 2023Mapping of attention mechanisms to a generalized Potts modelcited by 19
  8. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  9. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  10. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  11. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  12. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 16
14 more
  1. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  2. 2021Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgettingcited by 14
  3. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
  4. 2023Strong inductive biases provably prevent harmless interpolationcited by 10
  5. 2022On the Universal Approximation Property of Deep Fully Convolutional Neural Networkscited by 7
  6. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  7. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  8. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  9. 2022CNNs Avoid Curse of Dimensionality by Learning on Patchescited by 3
  10. 2023Short-Term Memory Convolutionscited by 3
  11. 2022The impact of memory on learning sequence-to-sequence taskscited by 1
  12. 2022Inference of a Rumor's Source in the Independent Cascade Modelcited by 0
  13. 2023Kernels, Data & Physicscited by 0
  14. 2026Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Locality in Image Diffusion Models Emerges from Data Statistics
  2. 2026Differentiable Kernel Ridge Regression for Deep Learning Pipelines
  3. 2021Learning curves of generic features maps for realistic datasets with a teacher-student model
  4. 2025Localized Diffusion Models
  5. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting
  6. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods
  7. 2026What Visual Generators Need from Teachers: Rethinking Representation Alignment
  8. 2023Are Large Kernels Better Teachers than Transformers for ConvNets?