# Optimal Function Approximation with Relu Neural Networks

arXiv: https://arxiv.org/abs/1909.03731 · HTML: https://mlsanity.com/paper/1909.03731/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2019.

## Built on by (most cited first, 9 of 9)

- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [Curvature-Aware Training for Coordinate Networks](https://mlsanity.com/paper/2305.08552/) (2023)
- [Disparate Impact on Group Accuracy of Linearization for Private Inference](https://mlsanity.com/paper/2402.03629/) (2024)
- [Towards Optimally Weighted Physics-Informed Neural Networks in Ocean Modelling](https://mlsanity.com/paper/2106.08747/) (2021)
- [Pointwise convergence of Fourier series and deep neural network for the indicator function of d-dimensional ball](https://mlsanity.com/paper/2304.08172/) (2023)
- [SkelEx and BoundEx: Natural Visualization of ReLU Neural Networks](https://mlsanity.com/paper/2305.05562/) (2023)
- [Deep State-Space Model for Predicting Cryptocurrency Price](https://mlsanity.com/paper/2311.14731/) (2023)
- [Barycentric Neural Networks and Length-Weighted Persistent Entropy Loss: A Green Geometric and Topological Framework for Function Approximation](https://mlsanity.com/paper/2509.06694/) (2025)
- [An in-depth look at approximation via deep and narrow neural networks](https://mlsanity.com/paper/2510.07202/) (2025)

## Abstract (arXiv)

We consider in this paper the optimal approximations of convex univariate functions with feed-forward Relu neural networks. We are interested in the following question: what is the minimal approximation error given the number of approximating linear pieces? We establish the necessary and sufficient conditions and uniqueness of optimal approximations, and give lower and upper bounds of the optimal approximation errors. Relu neural network architectures are then presented to generate these optimal approximations. Finally, we propose an algorithm to find the optimal approximations, as well as prove its convergence and validate it with experimental results.