# Exponential ReLU Neural Network Approximation Rates for Point and Edge Singularities

arXiv: https://arxiv.org/abs/2010.12217 · HTML: https://mlsanity.com/paper/2010.12217/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equations](https://mlsanity.com/paper/2112.08125/) (2021)
- [Neural and spectral operator surrogates: unified construction and expression rate bounds](https://mlsanity.com/paper/2207.04950/) (2022)
- [Deep Learning in Deterministic Computational Mechanics](https://mlsanity.com/paper/2309.15421/) (2023)
- [On the Optimal Expressive Power of ReLU DNNs and Its Application in Approximation with Kolmogorov Superposition Theorem](https://mlsanity.com/paper/2308.05509/) (2023)
- [ReLU Deep Neural Networks from the Hierarchical Basis Perspective](https://mlsanity.com/paper/2105.04156/) (2021)
- [Learning the Hodgkin-Huxley Model with Operator Learning Techniques](https://mlsanity.com/paper/2406.02173/) (2024)
- [Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions](https://mlsanity.com/paper/2312.14276/) (2023)
- [Approximation results for Gradient Descent trained Shallow Neural Networks in 1d](https://mlsanity.com/paper/2209.08399/) (2022)
- [Learning Homogenization for Elliptic Operators](https://mlsanity.com/paper/2306.12006/) (2023)
- [Neural Snowflakes: Universal Latent Graph Inference via Trainable Latent Geometries](https://mlsanity.com/paper/2310.15003/) (2023)
- [Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals](https://mlsanity.com/paper/2310.19603/) (2023)
- [A short tour of operator learning theory: Convergence rates, statistical limits, and open questions](https://mlsanity.com/paper/2603.00819/) (2026)
- [Deep ReLU networks and high-order finite element methods II: Chebyshev emulation](https://mlsanity.com/paper/2310.07261/) (2023)
- [Exponential Expressivity of ReLUk Neural Networks on Gevrey Classes with Point Singularities](https://mlsanity.com/paper/2403.02035/) (2024)
- [A Low Rank Neural Representation of Entropy Solutions](https://mlsanity.com/paper/2406.05694/) (2024)
- [Limitations of neural network training due to numerical instability of backpropagation](https://mlsanity.com/paper/2210.00805/) (2022)
- [Approximation Results for Gradient Descent trained Neural Networks](https://mlsanity.com/paper/2309.04860/) (2023)
- [Neural Networks for Singular Perturbations](https://mlsanity.com/paper/2401.06656/) (2024)
- [On the algorithmic construction of deep ReLU networks](https://mlsanity.com/paper/2506.19104/) (2025)
- [Learning Low Rank Neural Representations of Hyperbolic Wave Dynamics from Data](https://mlsanity.com/paper/2510.25123/) (2025)

## Abstract (arXiv)

We prove exponential expressivity with stable ReLU Neural Networks (ReLU NNs) in H1(Omega) for weighted analytic function classes in certain polytopal domains Omega, in space dimension d=2,3. Functions in these classes are locally analytic on open subdomains Dsubset Omega, but may exhibit isolated point singularities in the interior of Omega or corner and edge singularities at the boundary partial Omega. The exponential expression rate bounds proved here imply uniform exponential expressivity by ReLU NNs of solution families for several elliptic boundary and eigenvalue problems with analytic data. The exponential approximation rates are shown to hold in space dimension d = 2 on Lipschitz polygons with straight sides, and in space dimension d=3 on Fichera-type polyhedral domains with plane faces. The constructive proofs indicate in particular that NN depth and size increase poly-logarithmically with respect to the target NN approximation accuracy varepsilon>0 in H1(Omega). The results cover in particular solution sets of linear, second order elliptic PDEs with analytic data and certain nonlinear elliptic eigenvalue problems with analytic nonlinearities and singular, weighted analytic potentials as arise in electron structure models. In the latter case, the functions correspond to electron densities that exhibit isolated point singularities at the positions of the nuclei. Our findings provide in particular mathematical foundation of recently reported, successful uses of deep neural networks in variational electron structure algorithms.