mlsanityData to 14 Oct 2026

Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms

arXiv 2508.05141 ↗First posted 7 Aug 2025cs.LG cs.NA math.NAcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper establishes a comprehensive approximation result for deep fully-connected neural networks with commonly-used and general activation functions in Sobolev spaces Wn,∞, with errors measured in the Wm,p-norm for m < n and 1≤ p ≤ ∞. The derived rates surpass those of classical numerical approximation techniques, such as finite element and spectral methods, exhibiting a phenomenon we refer to as super-convergence. Our analysis shows that deep networks with general activations can approximate weak solutions of partial differential equations (PDEs) with superior accuracy compared to traditional numerical methods at the approximation level. Furthermore, this work closes a significant gap in the error-estimation theory for neural-network-based approaches to PDEs, offering a unified theoretical foundation for their use in scientific computing.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

27 of 52 references.

References · 27

  1. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 583
  2. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  3. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  4. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 191
  5. 2018Universality of Deep Convolutional Neural Networkscited by 176
  6. 2020Deep Network Approximation for Smooth Functionscited by 164
  7. 2019Deep Network Approximation Characterized by Number of Neuronscited by 122
  8. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 96
  9. 2019Error bounds for approximations with deep ReLU neural networks in Ws,p normscited by 95
  10. 2019The phase diagram of approximation rates for deep neural networkscited by 88
  11. 2021Sharp Bounds on the Approximation Rates, Metric Entropy, and n-widths of Shallow Neural Networkscited by 86
  12. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
15 more
  1. 2021On the approximation of functions by tanh neural networkscited by 81
  2. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  3. 2021Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEscited by 57
  4. 2019Nonlinear Approximation via Compositionscited by 38
  5. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  6. 2023Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivativescited by 30
  7. 2020Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernelscited by 19
  8. 2021Approximation Properties of Deep ReLU CNNscited by 15
  9. 2021Error Estimates for the Deep Ritz Method with Boundary Penaltycited by 14
  10. 2024Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Losscited by 14
  11. 2020Approximations with deep neural networks in Sobolev time-spacecited by 13
  12. 2023Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activationcited by 9
  13. 2020Soft-Root-Sign Activation Functioncited by 8
  14. 2022Approximation of Functionals by Neural Network without Curse of Dimensionalitycited by 8
  15. 2023Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensionscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spacescited by 3
  2. 2026Geometric regularization of autoencoders via observed stochastic dynamicscited by 2
  3. 2025Statistical Learning Guarantees for Group-Invariant Barron Functionscited by 1
  4. 2026Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximationscited by 0
  5. 2026Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activationscited by 0
  6. 2026Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Widthcited by 0