mlsanityData to 7 Oct 2026

Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attention

arXiv 2512.20562 ↗First posted 23 Dec 2025stat.ML cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the problem of learning a low-degree spherical polynomial of degree ℓ0 = Theta(1) ≥ 1 defined on the unit sphere in RRd by training an over-parameterized two-layer neural network (NN) with channel attention in this paper. Our main result is the significantly improved sample complexity for learning such low-degree polynomials. We show that, for any regression risk eps ∈ (0,1), a carefully designed two-layer NN with channel attention and finite width trained by the vanilla gradient descent (GD) requires the lowest sample complexity of n asymp Theta(dℓ0/eps) with high probability, in contrast with the representative sample complexity Thetapthdℓ0 maxseteps-2,log d, where n is the training data size. Moreover, such sample complexity is not improvable since the trained network renders a sharp rate of the nonparametric regression risk of the order Theta(dℓ0/n) with high probability. On the other hand, the minimax optimal rate for the regression risk with a kernel of rank Theta(dℓ0) is Theta(dℓ0/n), so that the rate of the nonparametric regression risk of the network trained by GD is minimax optimal. Training the two-layer NN with channel attention proceeds in two stages: (1) a provable learnable channel selection algorithm, as a learnable harmonic-degree selection process, identifies the ground truth channel number in the target function, ℓ0, from L ≥ ℓ0 channels in the first-layer activation; (2) the second layer is trained by standard GD using the selected channels. To the best of our knowledge, this is the first time a minimax optimal risk bound is obtained by training an over-parameterized but finite-width neural network with feature learning capability to learn low-degree spherical polynomials.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

38 of 57 references.

References · 38

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2018On the Spectral Bias of Neural Networkscited by 1,321
  4. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  8. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 572
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  11. 2021Random Feature Attentioncited by 350
  12. 2021XCiT: Cross-Covariance Image Transformerscited by 335
26 more
  1. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  2. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  3. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  4. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  5. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  6. 2019Linearized two-layers neural networks in high dimensioncited by 155
  7. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  8. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  9. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  10. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  11. 2013Early stopping and non-parametric regression: An optimal data-dependent stopping rulecited by 86
  12. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  13. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  14. 2020Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Networkcited by 38
  15. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  16. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  17. 2023On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domainscited by 30
  18. 2019On Learning Over-parameterized Neural Networks: A Functional Approximation Perspectivecited by 28
  19. 2021Why Approximate Matrix Square Root Outperforms Accurate SVD in Global Covariance Pooling?cited by 24
  20. 2023Efficient Attention via Control Variatescited by 23
  21. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  22. 2015Minimax Optimal Rates of Estimation in High Dimensional Additive Models: Universal Phase Transitioncited by 13
  23. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  24. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  25. 2024Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometrycited by 9
  26. 2024Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regressioncited by 2

Built on by

Not cited yet.