# Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

arXiv: https://arxiv.org/abs/2206.09527 · HTML: https://mlsanity.com/paper/2206.09527/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Builds on (most cited first, 22 of 22)

- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://mlsanity.com/paper/1708.02691/) (2017)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [The phase diagram of approximation rates for deep neural networks](https://mlsanity.com/paper/1906.09477/) (2019)
- [Optimal Control Via Neural Networks: A Convex Approach](https://mlsanity.com/paper/1805.11835/) (2018)
- [Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with  ℓ1  and  ℓ0  Controls](https://mlsanity.com/paper/1607.07819/) (2016)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [Neural Network Approximation: Three Hidden Layers Are Enough](https://mlsanity.com/paper/2010.14075/) (2020)
- [Approximation spaces of deep neural networks](https://mlsanity.com/paper/1905.01208/) (2019)
- [Numerical Solution of the Parametric Diffusion Equation by Deep Neural Networks](https://mlsanity.com/paper/2004.12131/) (2020)
- [Approximating smooth functions by deep neural networks with sigmoid activation function](https://mlsanity.com/paper/2010.04596/) (2020)
- [A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Finding](https://mlsanity.com/paper/2104.03270/) (2021)
- [Approximations with deep neural networks in Sobolev time-space](https://mlsanity.com/paper/2101.06115/) (2020)
- [PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Units](https://mlsanity.com/paper/1909.05136/) (2019)
- [Approximation of Smoothness Classes by Deep Rectifier Networks](https://mlsanity.com/paper/2007.15645/) (2020)
- [Evolved neuromorphic radar-based altitude controller for an autonomous open-source blimp](https://mlsanity.com/paper/2110.00646/) (2021)

## Built on by (most cited first, 29 of 29)

- [Enhancing Accuracy in Generative Models via Knowledge Transfer](https://mlsanity.com/paper/2405.16837/) (2024)
- [A prediction and behavioural analysis of machine learning methods for modelling travel mode choice](https://mlsanity.com/paper/2301.04404/) (2023)
- [Theoretical guarantees for neural control variates in MCMC](https://mlsanity.com/paper/2304.01111/) (2023)
- [Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections](https://mlsanity.com/paper/2504.15110/) (2025)
- [A Wasserstein perspective of Vanilla GANs](https://mlsanity.com/paper/2403.15312/) (2024)
- [On the estimation rate of Bayesian PINN for inverse problems](https://mlsanity.com/paper/2406.14808/) (2024)
- [Numerical and statistical analysis of NeuralODE with Runge-Kutta time integration](https://mlsanity.com/paper/2503.10729/) (2025)
- [Learning Brenier Potentials with Convex Generative Adversarial Neural Networks](https://mlsanity.com/paper/2504.19779/) (2025)
- [Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks](https://mlsanity.com/paper/2512.23643/) (2025)
- [Convergence of the Deep Galerkin Method for Finite State Mean Field Control Problems](https://mlsanity.com/paper/2405.13346/) (2024)
- [C1-approximation with rational functions and rational neural networks](https://mlsanity.com/paper/2508.19672/) (2025)
- [Expressive Power of Deep Networks on Manifolds: Simultaneous Approximation](https://mlsanity.com/paper/2509.09362/) (2025)
- [Approximation Capabilities of Feedforward Neural Networks with GELU Activations](https://mlsanity.com/paper/2512.21749/) (2025)
- [Learning Green's Function Efficiently Using Low-Rank Approximations](https://mlsanity.com/paper/2308.00350/) (2023)
- [Universality of reservoir systems with recurrent neural networks](https://mlsanity.com/paper/2403.01900/) (2024)
- [Improving Mixup Calibration with Wasserstein Distributionally Robust Optimization](https://mlsanity.com/paper/2506.17874/) (2025)
- [Stackelberg Coupling of Online Representation Learning and Reinforcement Learning](https://mlsanity.com/paper/2508.07452/) (2025)
- [Physically consistent model learning for reaction-diffusion systems](https://mlsanity.com/paper/2512.14240/) (2025)
- [A Quantitative Approximation Framework for Flow Distillation in Diffusion Models](https://mlsanity.com/paper/2606.03820/) (2026)
- [A statistical perspective on algorithm unrolling models for inverse problems](https://mlsanity.com/paper/2311.06395/) (2023)
- [On the growth of the parameters of approximating ReLU neural networks](https://mlsanity.com/paper/2406.14936/) (2024)
- [Scaling-based Data Augmentation for Generative Models and its Theoretical Extension](https://mlsanity.com/paper/2410.20780/) (2024)
- [Consistency of Learned Sparse Grid Quadrature Rules using NeuralODEs](https://mlsanity.com/paper/2507.01533/) (2025)
- [Regularity of Solutions to Beckmann's Parametric Optimal Transport](https://mlsanity.com/paper/2603.19755/) (2026)
- [Generative Adversarial Learning from Deterministic Processes](https://mlsanity.com/paper/2605.18425/) (2026)
- [Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations](https://mlsanity.com/paper/2606.05599/) (2026)
- [Nonparametric inference for density-dependent McKean--Vlasov diffusions](https://mlsanity.com/paper/2609.01166/) (2026)
- [Deep Barycentric Regression for Optimal Transport Map Estimation and its Statistical Optimality](https://mlsanity.com/paper/2609.06598/) (2026)
- [Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks](https://mlsanity.com/paper/2609.19937/) (2026)

## Abstract (arXiv)

This paper investigates the approximation properties of deep neural networks with piecewise-polynomial activation functions. We derive the required depth, width, and sparsity of a deep neural network to approximate any H"older smooth function up to a given approximation error in H"older norms in such a way that all weights of this neural network are bounded by 1. The latter feature is essential to control generalization errors in many statistical and machine learning applications.