mlsanityData to 7 Oct 2026

Activation degree thresholds and expressiveness of polynomial neural networks

arXiv 2408.04569 ↗First posted 8 Aug 2024cs.LG cs.NE math.AGcited by 14

Not on the map. Why?

Abstract (arXiv)

We study the expressive power of deep polynomial neural networks through the geometry of their neurovariety. We introduce the notion of the activation degree threshold of a network architecture to express when the dimension of the neurovariety achieves its theoretical maximum. We prove the existence of the activation degree threshold for all polynomial neural networks without width-one bottlenecks and demonstrate a universal upper bound that is quadratic in the width of largest size. In doing so, we prove the high activation degree conjecture of Kileel, Trager, and Bruna. Certain structured architectures have exceptional activation degree thresholds, making them especially expressive in the sense of their neurovariety dimension. In this direction, we prove that polynomial neural networks with equi-width architectures are maximally expressive by showing their activation degree threshold is one.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 24 of its 45 references that are in the map. First posted on arXiv 8 Aug 2024.

Builds on

24 of 45 references.

References · 24

  1. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  2. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  3. 2016On the Expressive Power of Deep Neural Networkscited by 302
  4. 2016Benefits of depth in neural networkscited by 248
  5. 2016Deep Learning without Poor Local Minimacited by 233
  6. 2019Universal Approximation with Deep Narrow Networkscited by 198
  7. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  8. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 120
  9. 2018Tropical Geometry of Deep Neural Networkscited by 92
  10. 2020Rational neural networkscited by 75
  11. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
  12. 2016Deep vs. shallow networks : An approximation theory perspectivecited by 63
12 more
  1. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 55
  2. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  3. 2018A Tropical Approach to Neural Networks with Piecewise Linear Activationscited by 26
  4. 2020On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspectivecited by 25
  5. 2024The Real Tropical Geometry of Neural Networkscited by 23
  6. 2024Geometry of Polynomial Neural Networkscited by 21
  7. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
  8. 2018Neural Networks Should Be Wide Enough to Learn Disconnected Decision Regionscited by 17
  9. 2021PolyNet: Polynomial Neural Network for 3D Shape Recognition with PolyShape Representationcited by 9
  10. 2022Local and global topological complexity measures OF ReLU neural network functionscited by 8
  11. 2023Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groupscited by 5
  12. 2019Learning Algebraic Models of Quantum Entanglementcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

  1. 2025Symmetry in Neural Network Parameter Spacescited by 31
  2. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  3. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  4. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  5. 2025The Alexander-Hirschowitz theorem for neurovarietiescited by 6
  6. 2026Most ReLU Networks Admit Identifiable Parameterscited by 5
  7. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  8. 2025Toric geometry of ReLU neural networkscited by 2
  9. 2026The Geometry of Polynomial Group Convolutional Neural Networkscited by 2
  10. 2026A Complete Symmetry Classification of Shallow ReLU Networkscited by 2
  11. 2026The Symmetries of Three-Layer ReLU Networkscited by 2
  12. 2026Algebraic Invariants of Lightning Self-Attentioncited by 1
2 more
  1. 2026Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defectscited by 1
  2. 2026It's Much Easier for Neural Networks to learn Game of Life Dynamics with the Right Activation Function: Polynomial Kolmogorov-Arnold Networkscited by 1