# Neural Network with Unbounded Activation Functions is Universal Approximator

arXiv: https://arxiv.org/abs/1505.03654 · HTML: https://mlsanity.com/paper/1505.03654/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2015.

## Built on by (most cited first, 30 of 63)

- [LaTeRF: Label and Text Driven Object Radiance Fields](https://mlsanity.com/paper/2207.01583/) (2022)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions](https://mlsanity.com/paper/2107.00896/) (2021)
- [Deep Reinforcement Learning for Adaptive Mesh Refinement](https://mlsanity.com/paper/2209.12351/) (2022)
- [Ridges, Neural Networks, and the Radon Transform](https://mlsanity.com/paper/2203.02543/) (2022)
- [The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks](https://mlsanity.com/paper/2306.17499/) (2023)
- [On the Complexity of Learning Sparse Functions with Statistical and Gradient Queries](https://mlsanity.com/paper/2407.05622/) (2024)
- [Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysis](https://mlsanity.com/paper/2203.01631/) (2022)
- [Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks](https://mlsanity.com/paper/2108.12943/) (2021)
- [Universal approximation property of Banach space-valued random feature models including random neural networks](https://mlsanity.com/paper/2312.08410/) (2023)
- [Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL](https://mlsanity.com/paper/2209.09845/) (2022)
- [Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups](https://mlsanity.com/paper/2205.14819/) (2022)
- [A Simple Approach to Automated Spectral Clustering](https://mlsanity.com/paper/2107.12183/) (2021)
- [C^*-algebra Net: A New Approach Generalizing Neural Network Parameters to C^*-algebra](https://mlsanity.com/paper/2206.09513/) (2022)
- [Universal Approximation Theorem for Neural Networks](https://mlsanity.com/paper/2102.10993/) (2021)
- [Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space](https://mlsanity.com/paper/2106.04770/) (2021)
- [Universal approximation and model compression for radial neural networks](https://mlsanity.com/paper/2107.02550/) (2021)
- [Universal approximation results for neural networks with non-polynomial activation function over non-compact domains](https://mlsanity.com/paper/2410.14759/) (2024)
- [CDiNN -Convex Difference Neural Networks](https://mlsanity.com/paper/2103.17231/) (2021)
- [Deep Algorithm Unrolling for Biomedical Imaging](https://mlsanity.com/paper/2108.06637/) (2021)
- [Solving stochastic partial differential equations using neural networks in the Wiener chaos expansion](https://mlsanity.com/paper/2411.03384/) (2024)
- [State and Topology Estimation for Unobservable Distribution Systems using Deep Neural Networks](https://mlsanity.com/paper/2104.07208/) (2021)
- [Quantum Ridgelet Transform: Winning Lottery Ticket of Neural Networks with Quantum Computation](https://mlsanity.com/paper/2301.11936/) (2023)
- [From Kernel Methods to Neural Networks: A Unifying Variational Formulation](https://mlsanity.com/paper/2206.14625/) (2022)
- [Information Removal at the bottleneck in Deep Neural Networks](https://mlsanity.com/paper/2210.00891/) (2022)
- [Noncommutative C^*-algebra Net: Learning Neural Networks with Powerful Product Structure in C^*-algebra](https://mlsanity.com/paper/2302.01191/) (2023)
- [Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Control](https://mlsanity.com/paper/2403.17142/) (2024)
- [Autoencoder-driven Spiral Representation Learning for Gravitational Wave Surrogate Modelling](https://mlsanity.com/paper/2107.04312/) (2021)
- [Deep learning fluid flow reconstruction around arbitrary two-dimensional objects from sparse sensors using conformal mappings](https://mlsanity.com/paper/2202.03798/) (2022)
- [Disentangling private classes through regularization](https://mlsanity.com/paper/2207.02000/) (2022)

## Abstract (arXiv)

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.