# PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Units

arXiv: https://arxiv.org/abs/1909.05136 · HTML: https://mlsanity.com/paper/1909.05136/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 12 of 12)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations](https://mlsanity.com/paper/2206.09527/) (2022)
- [PowerMLP: An Efficient Version of KAN](https://mlsanity.com/paper/2412.13571/) (2024)
- [On the Geometry and Optimization of Polynomial Convolutional Networks](https://mlsanity.com/paper/2410.00722/) (2024)
- [Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression](https://mlsanity.com/paper/2305.00608/) (2023)
- [De Rham compatible Deep Neural Network FEM](https://mlsanity.com/paper/2201.05395/) (2022)
- [Theoretical guarantees for neural control variates in MCMC](https://mlsanity.com/paper/2304.01111/) (2023)
- [Estimation of Non-Crossing Quantile Regression Process with Deep ReQU Neural Networks](https://mlsanity.com/paper/2207.10442/) (2022)
- [Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation](https://mlsanity.com/paper/2406.02336/) (2024)
- [Why Rectified Power Unit Networks Fail and How to Improve It: An Effective Field Theory Perspective](https://mlsanity.com/paper/2408.02697/) (2024)
- [On Explicit Super-Expressive Approximation for Neural Networks](https://mlsanity.com/paper/2607.06781/) (2026)
- [Rational Neural Network Controllers](https://mlsanity.com/paper/2307.06287/) (2023)

## Abstract (arXiv)

Deep neural network with rectified linear units (ReLU) is getting more and more popular recently. However, the derivatives of the function represented by a ReLU network are not continuous, which limit the usage of ReLU network to situations only when smoothness is not required. In this paper, we construct deep neural networks with rectified power units (RePU), which can give better approximations for smooth functions. Optimal algorithms are proposed to explicitly build neural networks with sparsely connected RePUs, which we call PowerNets, to represent polynomials with no approximation error. For general smooth functions, we first project the function to their polynomial approximations, then use the proposed algorithms to construct corresponding PowerNets. Thus, the error of best polynomial approximation provides an upper bound of the best RePU network approximation error. For smooth functions in higher dimensional Sobolev spaces, we use fast spectral transforms for tensor-product grid and sparse grid discretization to get polynomial approximations. Our constructive algorithms show clearly a close connection between spectral methods and deep neural networks: a PowerNet with n layers can exactly represent polynomials up to degree sn, where s is the power of RePUs. The proposed PowerNets have potential applications in the situations where high-accuracy is desired or smoothness is required.