# On Approximation Capabilities of ReLU Activation and Softmax Output Layer in Neural Networks

arXiv: https://arxiv.org/abs/2002.04060 · HTML: https://mlsanity.com/paper/2002.04060/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 5 of 5)

- [A Latent Space Theory for Emergent Abilities in Large Language Models](https://mlsanity.com/paper/2304.09960/) (2023)
- [S++: A Fast and Deployable Secure-Computation Framework for Privacy-Preserving Neural Network Training](https://mlsanity.com/paper/2101.12078/) (2021)
- [Neural-g: A Deep Learning Framework for Mixing Density Estimation](https://mlsanity.com/paper/2406.05986/) (2024)
- [Hybrid quantum-classical neural network for sentiment analysis](https://mlsanity.com/paper/2607.01943/) (2026)
- [Learning to Price Electricity for Optimal Demand Response](https://mlsanity.com/paper/2610.00755/) (2026)

## Abstract (arXiv)

In this paper, we have extended the well-established universal approximator theory to neural networks that use the unbounded ReLU activation function and a nonlinear softmax output layer. We have proved that a sufficiently large neural network using the ReLU activation function can approximate any function in L1 up to any arbitrary precision. Moreover, our theoretical results have shown that a large enough neural network using a nonlinear softmax output layer can also approximate any indicator function in L1, which is equivalent to mutually-exclusive class labels in any realistic multiple-class pattern classification problems. To the best of our knowledge, this work is the first theoretical justification for using the softmax output layers in neural networks for pattern classification.