mlsanityData to 7 Oct 2026

On the Expressive Power of Deep Polynomial Neural Networks

arXiv 1905.12207 ↗First posted 29 May 2019cs.LG cs.NE math.AGcited by 65

Not on the map. Why?

Abstract (arXiv)

We study deep neural networks with polynomial activations, particularly their expressive power. For a fixed architecture and activation degree, a polynomial neural network defines an algebraic map from weights to polynomials. The image of this map is the functional space associated to the network, and it is an irreducible algebraic variety upon taking closure. This paper proposes the dimension of this variety as a precise measure of the expressive power of polynomial neural networks. We obtain several theoretical results regarding this dimension as a function of architecture, including an exact formula for high activation degrees, as well as upper and lower bounds on layer widths in order for deep polynomials networks to fill the ambient functional space. We also present computational evidence that it is profitable in terms of expressiveness for layer widths to increase monotonically and then decrease monotonically. Finally, we link our study to favorable optimization properties when training weights, and we draw intriguing connections with tensor and polynomial decompositions.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 8; 2024: 13; 2025: 11; 2026: 18.

65 papers.

Sort:

By area:

  1. 2021Model Complexity of Deep Learning: A Surveycited by 64
  2. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  3. 2021Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEscited by 42
  4. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  5. 2025Symmetry in Neural Network Parameter Spacescited by 31
  6. 2023Polynomial Neural Fields for Subband Decomposition and Manipulationcited by 22
  7. 2021Augmenting Deep Classifiers with Polynomial Neural Networkscited by 21
  8. 2024Geometry of Polynomial Neural Networkscited by 21
  9. 2025Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometrycited by 21
  10. 2021On Expressivity and Trainability of Quadratic Networkscited by 19
  11. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  12. 2023Function Space and Critical Points of Linear Convolutional Networkscited by 19
38 more
  1. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  2. 2021Optimal Gradient-based Algorithms for Non-concave Bandit Optimizationcited by 15
  3. 2024Activation degree thresholds and expressiveness of polynomial neural networkscited by 14
  4. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  5. 2024On the Geometry and Optimization of Polynomial Convolutional Networkscited by 11
  6. 2021Deep Learning Generalization and the Convex Hull of Training Setscited by 10
  7. 2021On the Equivalence Between Temporal and Static Graph Representations for Observational Predictionscited by 10
  8. 2022DSGN++: Exploiting Visual-Spatial Relation for Stereo-based 3D Detectorscited by 10
  9. 2025Geometry and Optimization of Shallow Polynomial Networkscited by 10
  10. 2023Abstract Visual Reasoning: An Algebraic Approach for Solving Raven's Progressive Matricescited by 9
  11. 2024Polynomial Composition Activations: Unleashing the Dynamics of Large Language Modelscited by 9
  12. 2025Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networkscited by 9
  13. 2024Probabilistic Neural Circuitscited by 7
  14. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  15. 2023One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networkscited by 6
  16. 2023Polynomial Width is Sufficient for Set Representation with High-dimensional Featurescited by 6
  17. 2025The Alexander-Hirschowitz theorem for neurovarietiescited by 6
  18. 2023DeepBern-Nets: Taming the Complexity of Certifying Neural Networks using Bernstein Polynomial Activations and Precise Bound Propagationcited by 5
  19. 2023On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamicscited by 5
  20. 2022Theoretical guarantees for the advantage of GNNs over NNs in generalizing bandlimited functions on Euclidean cubescited by 4
  21. 2024HOIN: High-Order Implicit Neural Representationscited by 3
  22. 2024A Tensor Decomposition Perspective on Second-order RNNscited by 3
  23. 2024On the Expressive Power of Tree-Structured Probabilistic Circuitscited by 3
  24. 2026Robustness Verification of Polynomial Neural Networkscited by 3
  25. 2021Ladder Polynomial Neural Networkscited by 2
  26. 2021Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theoremcited by 2
  27. 2025A Training Framework for Optimal and Stable Training of Polynomial Neural Networkscited by 2
  28. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  29. 2026The Geometry of Polynomial Group Convolutional Neural Networkscited by 2
  30. 2026A Complete Symmetry Classification of Shallow ReLU Networkscited by 2
  31. 2026Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activationscited by 2
  32. 2022Deep Learning Generalization, Extrapolation, and Over-parameterizationcited by 1
  33. 2024Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximationcited by 1
  34. 2024Algebraic Representations for Faster Predictions in Convolutional Neural Networkscited by 1
  35. 2025Low-Rank Tensor Decompositions for the Theory of Neural Networkscited by 1
  36. 2026Rational Neural Networks have Expressivity Advantagescited by 1
  37. 2026Algebraic Invariants of Lightning Self-Attentioncited by 1
  38. 2026Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defectscited by 1

50 most cited of 65 All 65 (JSON)