# On the Expressive Power of Deep Polynomial Neural Networks

arXiv: https://arxiv.org/abs/1905.12207 · HTML: https://mlsanity.com/paper/1905.12207/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 30 of 66)

- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs](https://mlsanity.com/paper/2101.08366/) (2021)
- [The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning](https://mlsanity.com/paper/2205.06226/) (2022)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Polynomial Neural Fields for Subband Decomposition and Manipulation](https://mlsanity.com/paper/2302.04862/) (2023)
- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [Geometry of Polynomial Neural Networks](https://mlsanity.com/paper/2402.00949/) (2024)
- [Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometry](https://mlsanity.com/paper/2501.18915/) (2025)
- [On Expressivity and Trainability of Quadratic Networks](https://mlsanity.com/paper/2110.06081/) (2021)
- [The Spectral Bias of Polynomial Neural Networks](https://mlsanity.com/paper/2202.13473/) (2022)
- [Function Space and Critical Points of Linear Convolutional Networks](https://mlsanity.com/paper/2304.05752/) (2023)
- [Geometry of Lightning Self-Attention: Identifiability and Dimension](https://mlsanity.com/paper/2408.17221/) (2024)
- [Optimal Gradient-based Algorithms for Non-concave Bandit Optimization](https://mlsanity.com/paper/2107.04518/) (2021)
- [Activation degree thresholds and expressiveness of polynomial neural networks](https://mlsanity.com/paper/2408.04569/) (2024)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [On the Geometry and Optimization of Polynomial Convolutional Networks](https://mlsanity.com/paper/2410.00722/) (2024)
- [Deep Learning Generalization and the Convex Hull of Training Sets](https://mlsanity.com/paper/2101.09849/) (2021)
- [On the Equivalence Between Temporal and Static Graph Representations for Observational Predictions](https://mlsanity.com/paper/2103.07016/) (2021)
- [DSGN++: Exploiting Visual-Spatial Relation for Stereo-based 3D Detectors](https://mlsanity.com/paper/2204.03039/) (2022)
- [Geometry and Optimization of Shallow Polynomial Networks](https://mlsanity.com/paper/2501.06074/) (2025)
- [Abstract Visual Reasoning: An Algebraic Approach for Solving Raven's Progressive Matrices](https://mlsanity.com/paper/2303.11730/) (2023)
- [Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models](https://mlsanity.com/paper/2411.03884/) (2024)
- [Learning on a Razor's Edge: Identifiability and Singularity of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11846/) (2025)
- [Probabilistic Neural Circuits](https://mlsanity.com/paper/2403.06235/) (2024)
- [Identifiable Equivariant Networks are Layerwise Equivariant](https://mlsanity.com/paper/2601.21645/) (2026)
- [One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks](https://mlsanity.com/paper/2303.06316/) (2023)
- [Polynomial Width is Sufficient for Set Representation with High-dimensional Features](https://mlsanity.com/paper/2307.04001/) (2023)
- [The Alexander-Hirschowitz theorem for neurovarieties](https://mlsanity.com/paper/2511.19703/) (2025)
- [DeepBern-Nets: Taming the Complexity of Certifying Neural Networks using Bernstein Polynomial Activations and Precise Bound Propagation](https://mlsanity.com/paper/2305.13508/) (2023)

## Abstract (arXiv)

We study deep neural networks with polynomial activations, particularly their expressive power. For a fixed architecture and activation degree, a polynomial neural network defines an algebraic map from weights to polynomials. The image of this map is the functional space associated to the network, and it is an irreducible algebraic variety upon taking closure. This paper proposes the dimension of this variety as a precise measure of the expressive power of polynomial neural networks. We obtain several theoretical results regarding this dimension as a function of architecture, including an exact formula for high activation degrees, as well as upper and lower bounds on layer widths in order for deep polynomials networks to fill the ambient functional space. We also present computational evidence that it is profitable in terms of expressiveness for layer widths to increase monotonically and then decrease monotonically. Finally, we link our study to favorable optimization properties when training weights, and we draw intriguing connections with tensor and polynomial decompositions.