mlsanityData to 14 Oct 2026

Neural Network with Unbounded Activation Functions is Universal Approximator

arXiv 1505.03654 ↗First posted 14 May 2015cs.NE cs.LG math.FAcited by 63

Not on the map. Why?

Abstract (arXiv)

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 May 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 17; 2023: 9; 2024: 13; 2025: 5; 2026: 6.

63 papers.

Sort:

By area:

  1. 2022LaTeRF: Label and Text Driven Object Radiance Fieldscited by 38
  2. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  3. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  4. 2022Deep Reinforcement Learning for Adaptive Mesh Refinementcited by 20
  5. 2022Ridges, Neural Networks, and the Radon Transformcited by 18
  6. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  7. 2024On the Complexity of Learning Sparse Functions with Statistical and Gradient Queriescited by 15
  8. 2022Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysiscited by 14
  9. 2021Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networkscited by 13
  10. 2023Universal approximation property of Banach space-valued random feature models including random neural networkscited by 13
  11. 2022Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARLcited by 12
  12. 2022Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groupscited by 11
38 more
  1. 2021A Simple Approach to Automated Spectral Clusteringcited by 10
  2. 2022C^*-algebra Net: A New Approach Generalizing Neural Network Parameters to C^*-algebracited by 9
  3. 2021Universal Approximation Theorem for Neural Networkscited by 7
  4. 2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
  5. 2021Universal approximation and model compression for radial neural networkscited by 7
  6. 2024Universal approximation results for neural networks with non-polynomial activation function over non-compact domainscited by 7
  7. 2021CDiNN -Convex Difference Neural Networkscited by 6
  8. 2021Deep Algorithm Unrolling for Biomedical Imagingcited by 6
  9. 2024Solving stochastic partial differential equations using neural networks in the Wiener chaos expansioncited by 6
  10. 2021State and Topology Estimation for Unobservable Distribution Systems using Deep Neural Networkscited by 5
  11. 2023Quantum Ridgelet Transform: Winning Lottery Ticket of Neural Networks with Quantum Computationcited by 5
  12. 2022From Kernel Methods to Neural Networks: A Unifying Variational Formulationcited by 4
  13. 2022Information Removal at the bottleneck in Deep Neural Networkscited by 4
  14. 2023Noncommutative C^*-algebra Net: Learning Neural Networks with Powerful Product Structure in C^*-algebracited by 4
  15. 2024Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Controlcited by 4
  16. 2021Autoencoder-driven Spiral Representation Learning for Gravitational Wave Surrogate Modellingcited by 3
  17. 2022Deep learning fluid flow reconstruction around arbitrary two-dimensional objects from sparse sensors using conformal mappingscited by 3
  18. 2022Disentangling private classes through regularizationcited by 3
  19. 2023Universal approximation with complex-valued deep narrow neural networkscited by 3
  20. 2022L4KDE: Learning for KinoDynamic Tree Expansioncited by 2
  21. 2022A Feedforward Unitary Equivariant Neural Networkcited by 2
  22. 2024A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networkscited by 2
  23. 2024Neural Fractional Differential Equationscited by 2
  24. 2024Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machinescited by 2
  25. 2024Interpreting Deep Neural Network-Based Receiver Under Varying Signal-To-Noise Ratioscited by 2
  26. 2024Nonuniform random feature models using derivative informationcited by 2
  27. 2023Out-of-Distribution Optimality of Invariant Risk Minimizationcited by 1
  28. 2023Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networkscited by 1
  29. 2023Analytical Verification of Performance of Deep Neural Network Based Time-Synchronized Distribution System State Estimationcited by 1
  30. 2024From Robustness to Improved Generalization and Calibration in Pre-trained Language Modelscited by 1
  31. 2024Universal approximation property of ODENet and ResNet with a single activation functioncited by 1
  32. 2025Scalable Causal Discovery from Recursive Nonlinear Data via Truncated Basis Function Scores and Testscited by 1
  33. 2025Bayesian neural networks with interpretable priors from Mercer kernelscited by 1
  34. 2021Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximationcited by 0
  35. 2021Learning to Control using Image Feedbackcited by 0
  36. 2021Cooperative Deep Q-learning Framework for Environments Providing Image Feedbackcited by 0
  37. 2022The (Un)Scalability of Heuristic Approximators for NP-Hard Search Problemscited by 0
  38. 2022Optimization-Informed Neural Networkscited by 0

50 most cited of 63 All 63 (JSON)