# SUPN: Shallow Universal Polynomial Networks for Efficient Function Approximation

arXiv: https://arxiv.org/abs/2511.21414 · HTML: https://mlsanity.com/paper/2511.21414/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 70 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations](https://mlsanity.com/paper/2107.07871/) (2021)
- [A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks](https://mlsanity.com/paper/2406.02917/) (2024)
- [DeepOKAN: Deep Operator Network Based on Kolmogorov Arnold Networks for Mechanics Problems](https://mlsanity.com/paper/2405.19143/) (2024)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [On the Expressive Power of Deep Polynomial Neural Networks](https://mlsanity.com/paper/1905.12207/) (2019)
- [Neural Network with Unbounded Activation Functions is Universal Approximator](https://mlsanity.com/paper/1505.03654/) (2015)
- [Deep Polynomial Neural Networks](https://mlsanity.com/paper/2006.13026/) (2020)
- [Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpoint](https://mlsanity.com/paper/1912.04862/) (2019)
- [Exact and Inexact Subsampled Newton Methods for Optimization](https://mlsanity.com/paper/1609.08502/) (2016)
- [Collapse of Deep and Narrow Neural Nets](https://mlsanity.com/paper/1808.04947/) (2018)
- [Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study](https://mlsanity.com/paper/1708.07827/) (2017)
- [A novel meta-learning initialization method for physics-informed neural networks](https://mlsanity.com/paper/2107.10991/) (2021)
- [A Metalearning Approach for Physics-Informed Neural Networks (PINNs): Application to Parameterized PDEs](https://mlsanity.com/paper/2110.13361/) (2021)
- [Geometry of Polynomial Neural Networks](https://mlsanity.com/paper/2402.00949/) (2024)
- [Improved weight initialization for deep and narrow feedforward neural network](https://mlsanity.com/paper/2311.03733/) (2023)
- [Growing axons: greedy learning of neural networks with application to function approximation](https://mlsanity.com/paper/1910.12686/) (2019)
- [Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training](https://mlsanity.com/paper/1905.06738/) (2019)
- [Chebyshev Feature Neural Network for Accurate Function Approximation](https://mlsanity.com/paper/2409.19135/) (2024)
- [Leveraging KANs for Expedient Training of Multichannel MLPs via Preconditioning and Geometric Refinement](https://mlsanity.com/paper/2505.18131/) (2025)

## Abstract (arXiv)

Deep neural networks (DNNs) and Kolmogorov-Arnold networks (KANs) are flexible function approximators, but their accuracy can require substantial network depth, width, and trainable parameter counts. Beyond making the resulting networks less transparent, large parameter spaces can increase training cost and produce solutions that are sensitive to optimization and parameter initialization, leading to variability in generalization error. For these reasons, we propose Shallow Universal Polynomial Networks (SUPNs), which retain a single hidden layer while increasing the expressivity of each unit through a learnable multivariate polynomial inside each activation, which we refer to as a polynomial pre-activation. This construction trades architectural depth and width for greater expressivity within each hidden unit, reducing parameter counts and training cost while preserving a shallow computational graph. We establish two complementary universal approximation results: SUPNs are universal by increasing network width at fixed polynomial complexity, as in classical shallow-network approximation, and by increasing polynomial complexity at fixed finite width. We further prove that finite SUPNs can approach the best approximation attainable in a prescribed polynomial space to arbitrary relative tolerance, derive explicit quasi-optimal parameters, and establish corresponding convergence rates. We empirically evaluate SUPNs against DNNs, KANs, and polynomial projection on smooth, nonsmooth, and discontinuous function-approximation problems in one, two, and ten dimensions using more than 16,000 trained models. Across the target functions studied, SUPNs frequently achieve lower approximation error and variability than DNNs and KANs at comparable parameter counts while retaining lower training cost, supporting their use as efficient function approximators when the target structure is not known a priori.