# The limitation of neural nets for approximation and optimization

arXiv: https://arxiv.org/abs/2311.12253 · HTML: https://mlsanity.com/paper/2311.12253/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 77 references that are in the map.
- First posted on arXiv 21 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization](https://mlsanity.com/paper/1905.01332/) (2019)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Discovering Parametric Activation Functions](https://mlsanity.com/paper/2006.03179/) (2020)
- [Learning Combinations of Activation Functions](https://mlsanity.com/paper/1801.09403/) (2018)
- [Scheduling Techniques for Liver Segmentation: ReduceLRonPlateau Vs OneCycleLR](https://mlsanity.com/paper/2202.06373/) (2022)
- [Learning activation functions from data using cubic spline interpolation](https://mlsanity.com/paper/1605.05509/) (2016)
- [Global Convergence Rate Analysis of a Generic Line Search Algorithm with Noise](https://mlsanity.com/paper/1910.04055/) (2019)
- [Learning to Learn by Zeroth-Order Oracle](https://mlsanity.com/paper/1910.09464/) (2019)
- [Derivative-Free Optimization of Noisy Functions via Quasi-Newton Methods](https://mlsanity.com/paper/1803.10173/) (2018)
- [Universal Activation Function For Machine Learning](https://mlsanity.com/paper/2011.03842/) (2020)
- [Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization](https://mlsanity.com/paper/1306.5729/) (2013)
- [Derivative-free methods for mixed-integer nonsmooth constrained optimization](https://mlsanity.com/paper/2107.00601/) (2021)
- [Full-low evaluation methods for derivative-free optimization](https://mlsanity.com/paper/2107.11908/) (2021)

## Built on by (most cited first, 1 of 1)

- [Comparison of derivative-free and gradient-based minimization for multi-objective compositional design of shape memory alloys](https://mlsanity.com/paper/2508.14127/) (2025)

## Abstract (arXiv)

We are interested in assessing the use of neural networks as surrogate models to approximate and minimize objective functions in optimization problems. While neural networks are widely used for machine learning tasks such as classification and regression, their application in solving optimization problems has been limited. Our study begins by determining the best activation function for approximating the objective functions of popular nonlinear optimization test problems, and the evidence provided shows that~SiLU has the best performance. We then analyze the accuracy of function value, gradient, and Hessian approximations for such objective functions obtained through interpolation/regression models and neural networks. When compared to interpolation/regression models, neural networks can deliver competitive zero- and first-order approximations (at a high training cost) but underperform on second-order approximation. However, it is shown that combining a neural net activation function with the natural basis for quadratic interpolation/regression can waive the necessity of including cross terms in the natural basis, leading to models with fewer parameters to determine. Lastly, we provide evidence that the performance of a state-of-the-art derivative-free optimization algorithm can hardly be improved when the gradient of an objective function is approximated using any of the surrogate models considered, including neural networks.