mlsanityData to 7 Oct 2026

Approximation Capabilities of Feedforward Neural Networks with GELU Activations

arXiv 2512.21749 ↗First posted 25 Dec 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We derive an approximation error bound that holds simultaneously for a function and all its derivatives up to any prescribed order. The bounds apply to elementary functions, including multivariate polynomials, the exponential function, and the reciprocal function, and are obtained using feedforward neural networks with the Gaussian Error Linear Unit (GELU) activation. In addition, we report the network size, weight magnitudes, and behavior at infinity. Our analysis begins with a constructive approximation of multiplication, where we prove the simultaneous validity of error bounds over domains of increasing size for a given approximator. Leveraging this result, we obtain approximation guarantees for division and the exponential function, ensuring that all higher-order derivatives of the resulting approximators remain globally bounded.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 14 of its 22 references that are in the map. First posted on arXiv 25 Dec 2025.

Builds on

14 of 22 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2023Diffusion Models are Minimax Optimal Distribution Estimatorscited by 159
  6. 2021On the approximation of functions by tanh neural networkscited by 81
  7. 2024Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensionscited by 60
  8. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  9. 2024Flow matching achieves almost minimax optimal convergencecited by 27
  10. 2023Breaking the Heavy-Tailed Noise Barrier in Stochastic Optimization Problemscited by 21
  11. 2025Generalization error bound for denoising score matching under relaxed manifold assumptioncited by 18
  12. 2023Noncompact uniform universal approximationcited by 7
2 more
  1. 2024Weighted Sobolev Approximation Rates for Neural Networks on Unbounded Domainscited by 5
  2. 2021Deep Learning in High Dimension: Neural Network Approximation of Analytic Functions in L2(ℝd,γd)cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networkscited by 3
  2. 2026Enforcing Dirichlet Boundary Conditions in Operator Learningcited by 1