mlsanityData to 14 Oct 2026

Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networks

arXiv 2112.05611 ↗First posted 10 Dec 2021cs.LG cs.AI stat.MLcited by 17

Not on the map. Why?

Abstract (arXiv)

Understanding the fundamental principles behind the massive success of neural networks is one of the most important open questions in deep learning. However, due to the highly complex nature of the problem, progress has been relatively slow. In this note, through the lens of infinite-width networks, a.k.a. neural kernels, we present one such principle resulting from hierarchical localities. It is well-known that the eigenstructure of infinite-width multilayer perceptrons (MLPs) depends solely on the concept frequency, which measures the order of interactions. We show that the topologies from deep convolutional networks (CNNs) restructure the associated eigenspaces into finer subspaces. In addition to frequency, the new structure also depends on the concept space, which measures the spatial distance among nonlinear interaction terms. The resulting fine-grained eigenstructure dramatically improves the network's learnability, empowering them to simultaneously model a much richer class of interactions, including Long-Range-Low-Frequency interactions, Short-Range-High-Frequency interactions, and various interpolations and extrapolations in-between. Additionally, model scaling can improve the resolutions of interpolations and extrapolations and, therefore, the network's learnability. Finally, we prove a sharp characterization of the generalization error for infinite-width CNNs of any depth in the high-dimensional setting. Two corollaries follow: (1) infinite-width deep CNNs can break the curse of dimensionality without losing their expressivity, and (2) scaling improves performance in both the finite and infinite data regimes.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 48 of its 82 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

48 of 82 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2018On the Spectral Bias of Neural Networkscited by 1,341
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  12. 2018On Lazy Training in Differentiable Programmingcited by 675
36 more
  1. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  2. 2017Deep Neural Networks as Gaussian Processescited by 622
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  4. 2021Explaining Neural Scaling Lawscited by 363
  5. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 345
  6. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  7. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  8. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  9. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  10. 2016Deep Information Propagationcited by 195
  11. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  12. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  13. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  14. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  15. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  16. 2019Linearized two-layers neural networks in high dimensioncited by 155
  17. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  18. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  19. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  20. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  21. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  22. 2021Learning with invariances in random features and kernel modelscited by 89
  23. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  24. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  25. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  26. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  27. 2020Neural Kernels Without Tangentscited by 56
  28. 2017Depth Separation for Neural Networkscited by 46
  29. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  30. 2020Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?cited by 34
  31. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  32. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  33. 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
  34. 2020Computational Separation Between Convolutional and Fully-Connected Networkscited by 20
  35. 2020Harmonic Decompositions of Convolutional Networkscited by 6
  36. 2021CausalX: Causal Explanations and Block Multilinear Factor Analysiscited by 3

Built on by

Cited by year

Citing papers per year: 2022: 9; 2023: 3; 2024: 5.

17 papers.

Sort:

By area:

  1. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  2. 2023Six Lectures on Linearized Neural Networkscited by 20
  3. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  4. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  5. 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 16
  6. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  7. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  8. 2022A Kernel Perspective of Skip Connections in Convolutional Networkscited by 12
  9. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
  10. 2023Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrumcited by 9
  11. 2022On the Universal Approximation Property of Deep Fully Convolutional Neural Networkscited by 7
  12. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
5 more
  1. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
  2. 2024Spectral complexity of deep neural networkscited by 3
  3. 2024Demystifying Spectral Bias on Real-World Datacited by 3
  4. 2022Testing predictions of representation cost theory with CNNscited by 0
  5. 2023Kernels, Data & Physicscited by 0