mlsanityData to 7 Oct 2026

On the approximation of functions by tanh neural networks

arXiv 2104.08938 ↗First posted 18 Apr 2021math.NA cs.LG cs.NAcited by 81

Not on the map. Why?

Abstract (arXiv)

We derive bounds on the error, in high-order Sobolev norms, incurred in the approximation of Sobolev-regular as well as analytic functions by neural networks with the hyperbolic tangent activation function. These bounds provide explicit estimates on the approximation error with respect to the size of the neural networks. We show that tanh neural networks with only two hidden layers suffice to approximate functions at comparable or better rates than much deeper ReLU neural networks.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 29 of its 75 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

29 of 75 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  4. 2016Error bounds for approximations with deep ReLU networkscited by 567
  5. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  6. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  7. 2017Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equationscited by 208
  8. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 188
  9. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  10. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  11. 2020Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEscited by 115
  12. 2016Why does deep and cheap learning work so well?cited by 100
17 more
  1. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  2. 2019A Theoretical Analysis of Deep Neural Networks and Parametric PDEscited by 74
  3. 2019Error bounds for deep ReLU networks using the Kolmogorov--Arnold superposition theoremcited by 49
  4. 2018Exponential Convergence of the Deep Neural Network Approximation for Analytic Functionscited by 38
  5. 2019Deep learning observables in computational fluid dynamicscited by 38
  6. 2018Collapse of Deep and Narrow Neural Netscited by 37
  7. 2017The power of deeper networks for expressing natural functionscited by 35
  8. 2020Physics Informed Neural Networks for Simulating Radiative Transfercited by 35
  9. 2019Smooth function approximation by deep neural networks with general activation functionscited by 29
  10. 2020Iterative Surrogate Model Optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networkscited by 26
  11. 2021Approximation capability of two hidden layer feedforward neural networks with fixed weightscited by 24
  12. 2020Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequencescited by 20
  13. 2017On the approximation by single hidden layer feedforward neural networks with fixed weightscited by 19
  14. 2019Full error analysis for the training of deep neural networkscited by 17
  15. 2020Higher-order Quasi-Monte Carlo Training of Deep Neural Networkscited by 12
  16. 2020Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisationcited by 7
  17. 2020The Representation Power of Neural Networks: Breaking the Curse of Dimensionalitycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 11; 2024: 18; 2025: 19; 2026: 16.

81 papers.

Sort:

By area:

  1. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
  2. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
  3. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 109
  4. 2022Generic bounds on the approximation error for physics-informed (and) operator learningcited by 65
  5. 2022Error estimates for physics informed neural networks approximating the Navier-Stokes equationscited by 63
  6. 2021Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEscited by 56
  7. 2021Symplectic Learning for Hamiltonian Neural Networkscited by 35
  8. 2023Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivativescited by 30
  9. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  10. 2022wPINNs: Weak Physics informed neural networks for approximating entropy solutions of hyperbolic conservation lawscited by 29
  11. 2022Variable-Input Deep Operator Networkscited by 24
  12. 2021Variational Physics Informed Neural Networks: the role of quadratures and test functionscited by 17
38 more
  1. 2021Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networkscited by 16
  2. 2022Instance-Dependent Generalization Bounds via Optimal Transportcited by 13
  3. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 10
  4. 2024Learning smooth functions in high dimensions: from sparse polynomials to deep neural networkscited by 7
  5. 2022Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?cited by 6
  6. 2025KANO: Kolmogorov-Arnold Neural Operatorcited by 6
  7. 2023Error convergence and engineering-guided hyperparameter search of PINNs: towards optimized I-FENN performancecited by 5
  8. 2023Entropy-dissipation Informed Neural Network for McKean-Vlasov Type PDEscited by 5
  9. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  10. 2025Deep Neural Networks with General Activations: Super-Convergence in Sobolev Normscited by 5
  11. 2023A prediction and behavioural analysis of machine learning methods for modelling travel mode choicecited by 4
  12. 2023Theoretical guarantees for neural control variates in MCMCcited by 4
  13. 2023Deep ReLU networks and high-order finite element methods II: Chebyshev emulationcited by 3
  14. 2023Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signalscited by 3
  15. 2024Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networkscited by 3
  16. 2025Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regressioncited by 3
  17. 2025Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networkscited by 3
  18. 2022Residual-Concatenate Neural Network with Deep Regularization Layers for Binary Classificationcited by 2
  19. 2022Error analysis for deep neural network approximations of parametric hyperbolic conservation lawscited by 2
  20. 2022The Mori-Zwanzig formulation of deep learningcited by 2
  21. 2023PINNs error estimates for nonlinear equations in ℝ-smooth Banach spacescited by 2
  22. 2023Physics-Informed Deep Learning of Rate-and-State Fault Frictioncited by 2
  23. 2024Error estimation for physics-informed neural networks with implicit Runge-Kutta methodscited by 2
  24. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Expertscited by 2
  25. 2024Convergence of the Deep Galerkin Method for Finite State Mean Field Control Problemscited by 2
  26. 2024On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processescited by 2
  27. 2024Physics-informed deep learning and compressive collocation for high-dimensional diffusion-reaction equations: practical existence theory and numericscited by 2
  28. 2024DiGRAF: Diffeomorphic Graph-Adaptive Activation Functioncited by 2
  29. 2025Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connectionscited by 2
  30. 2025Expressive Power of Deep Networks on Manifolds: Simultaneous Approximationcited by 2
  31. 2025Approximation Capabilities of Feedforward Neural Networks with GELU Activationscited by 2
  32. 2023Error Analysis of Physics-Informed Neural Networks for Approximating Dynamic PDEs of Second Order in Timecited by 1
  33. 2024Neural Networks for Singular Perturbationscited by 1
  34. 2024Error Estimation for Physics-informed Neural Networks Approximating Semilinear Wave Equationscited by 1
  35. 2024Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Methodcited by 1
  36. 2024Golden Ratio-Based Sufficient Dimension Reductioncited by 1
  37. 2025Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebracited by 1
  38. 2025Neural Measures for learning distributions of Random PDEscited by 1

50 most cited of 81 All 81 (JSON)