# Approximation Capabilities of Feedforward Neural Networks with GELU Activations

arXiv: https://arxiv.org/abs/2512.21749 · HTML: https://mlsanity.com/paper/2512.21749/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 25 Dec 2025.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Diffusion Models are Minimax Optimal Distribution Estimators](https://mlsanity.com/paper/2303.01861/) (2023)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions](https://mlsanity.com/paper/2409.18804/) (2024)
- [Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations](https://mlsanity.com/paper/2206.09527/) (2022)
- [Flow matching achieves almost minimax optimal convergence](https://mlsanity.com/paper/2405.20879/) (2024)
- [Breaking the Heavy-Tailed Noise Barrier in Stochastic Optimization Problems](https://mlsanity.com/paper/2311.04161/) (2023)
- [Generalization error bound for denoising score matching under relaxed manifold assumption](https://mlsanity.com/paper/2502.13662/) (2025)
- [Noncompact uniform universal approximation](https://mlsanity.com/paper/2308.03812/) (2023)
- [Weighted Sobolev Approximation Rates for Neural Networks on Unbounded Domains](https://mlsanity.com/paper/2411.04108/) (2024)
- [Deep Learning in High Dimension: Neural Network Approximation of Analytic Functions in L2(ℝd,γd)](https://mlsanity.com/paper/2111.07080/) (2021)

## Built on by (most cited first, 2 of 2)

- [Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks](https://mlsanity.com/paper/2512.23643/) (2025)
- [Enforcing Dirichlet Boundary Conditions in Operator Learning](https://mlsanity.com/paper/2608.27256/) (2026)

## Abstract (arXiv)

We derive an approximation error bound that holds simultaneously for a function and all its derivatives up to any prescribed order. The bounds apply to elementary functions, including multivariate polynomials, the exponential function, and the reciprocal function, and are obtained using feedforward neural networks with the Gaussian Error Linear Unit (GELU) activation. In addition, we report the network size, weight magnitudes, and behavior at infinity. Our analysis begins with a constructive approximation of multiplication, where we prove the simultaneous validity of error bounds over domains of increasing size for a given approximator. Leveraging this result, we obtain approximation guarantees for division and the exponential function, ensuring that all higher-order derivatives of the resulting approximators remain globally bounded.