mlsanityData to 7 Oct 2026

Geometry and Optimization of Shallow Polynomial Networks

arXiv 2501.06074 ↗First posted 10 Jan 2025cs.LG math.AGcited by 10

Not on the map. Why?

Abstract (arXiv)

We study shallow neural networks with monomial activations and output dimension one. The function space for these models can be identified with a set of symmetric tensors with bounded rank. We describe general features of these networks, focusing on the relationship between width and optimization. We then consider teacher-student problems, which can be viewed as problems of low-rank tensor approximation with respect to non-standard inner products that are induced by the data distribution. In this setting, we introduce a teacher-metric data discriminant which encodes the qualitative behavior of the optimization as a function of the training data distribution. Finally, we focus on networks with quadratic activations, presenting an in-depth analysis of the optimization landscape. In particular, we present a variation of the Eckart-Young Theorem characterizing all critical points and their Hessian signatures for teacher-student problems with quadratic networks and Gaussian training data.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

29 of 46 references.

References · 29

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  3. 2016Deep Learning without Poor Local Minimacited by 233
  4. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  5. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  6. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  7. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  8. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  9. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
  10. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  11. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  12. 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
17 more
  1. 2021Geometry of Linear Convolutional Networkscited by 27
  2. 2017On the Optimization Landscape of Tensor Decompositionscited by 21
  3. 2024Geometry of Polynomial Neural Networkscited by 21
  4. 2022The effect of smooth parametrizations on nonconvex optimization landscapescited by 20
  5. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  6. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  7. 2013The Euclidean distance degree of an algebraic varietycited by 15
  8. 2024Activation degree thresholds and expressiveness of polynomial neural networkscited by 14
  9. 2021Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networkscited by 12
  10. 2021Symmetry Breaking in Symmetric Tensor Decompositioncited by 10
  11. 2021Landscape analysis of an improved power method for tensor decompositioncited by 7
  12. 2022Tensor Moments of Gaussian Mixture Models: Theory and Applicationscited by 7
  13. 2022Moment Estimation for Nonparametric Mixture Models Through Implicit Tensor Decompositioncited by 4
  14. 2017Best rank-k approximations for tensors: generalizing Eckart-Youngcited by 3
  15. 2023Symmetry & Critical Points for Symmetric Tensor Decomposition Problemscited by 3
  16. 2019Avoiding Spurious Local Minima in Deep Quadratic Networkscited by 2
  17. 2024Symmetry & Critical Pointscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  2. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  3. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  4. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  5. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  6. 2025Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLUcited by 1
  7. 2025Critical Points of Degenerate Metrics on Algebraic Varieties: A Tale of Overparametrizationcited by 1
  8. 2026Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defectscited by 1
  9. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0
  10. 2026Algebraic Expressivity Certificates for Shallow Polynomial Neural Networkscited by 0