mlsanityData to 7 Oct 2026

Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent

arXiv 2205.07331 ↗First posted 15 May 2022math.NA cs.LG cs.NAcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the statistical limits in terms of Sobolev norms of gradient descent for solving inverse problem from randomly sampled noisy observations using a general class of objective functions. Our class of objective functions includes Sobolev training for kernel regression, Deep Ritz Methods (DRM), and Physics Informed Neural Networks (PINN) for solving elliptic partial differential equations (PDEs) as special cases. We consider a potentially infinite-dimensional parameterization of our model using a suitable Reproducing Kernel Hilbert Space and a continuous parameterization of problem hardness through the definition of kernel integral operators. We prove that gradient descent over this objective function can also achieve statistical optimality and the optimal number of passes over the data increases with sample size. Based on our theory, we explain an implicit acceleration of using a Sobolev norm as the objective function for training, inferring that the optimal number of epochs of DRM becomes larger than the number of PINN when both the data size and the hardness of tasks increase, although both DRM and PINN can achieve statistical optimality.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 45 of its 113 references that are in the map. First posted on arXiv 15 May 2022.

Builds on

45 of 113 references.

References · 45

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  4. 2018On the Spectral Bias of Neural Networkscited by 1,321
  5. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  8. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  9. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  10. 2017PDE-Net: Learning PDEs from Datacited by 304
  11. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  12. 2021Gradient-enhanced physics-informed neural networks for forward and inverse PDE problemscited by 220
33 more
  1. 2018PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Networkcited by 208
  2. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  3. 2017Sobolev Training for Neural Networkscited by 136
  4. 2021Solving and Learning Nonlinear PDEs with Gaussian Processescited by 128
  5. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 113
  6. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 83
  7. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  8. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  9. 2016Optimal Rates For Regularization Of Statistical Inverse Learning Problemscited by 68
  10. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  11. 2021Plugin Estimation of Smooth Optimal Transport Mapscited by 67
  12. 2018Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spacescited by 59
  13. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  14. 2021Convergence Rates for Learning Linear Operators from Noisy Datacited by 55
  15. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  16. 2020Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theorycited by 41
  17. 2021Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimalitycited by 40
  18. 2020Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Networkcited by 38
  19. 2020Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approachcited by 35
  20. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  21. 2020When Does Preconditioning Help or Hurt Generalization?cited by 30
  22. 2019Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordancecited by 28
  23. 2017Early stopping for kernel boosting algorithms: A general analysis with localized complexitiescited by 27
  24. 2020Deep neural network approximation for high-dimensional elliptic PDEs with boundary conditionscited by 25
  25. 2020The Statistical Complexity of Early-Stopped Mirror Descentcited by 19
  26. 2021A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equationscited by 18
  27. 2021Parametric Complexity Bounds for Approximating PDEs with Neural Networkscited by 18
  28. 2019Implicit Regularization of Accelerated Methods in Hilbert Spacescited by 14
  29. 2020Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learningcited by 14
  30. 2020Stochastic Gradient Descent in Hilbert Scales: Smoothness, Preconditioning and Earlier Stoppingcited by 5
  31. 2020Repulsive Curvescited by 4
  32. 2021Repulsive Surfacescited by 4
  33. 2020Gradient Descent in RKHS with Importance Labelingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 3; 2026: 2.

11 papers.

  1. 2022Minimax Optimal Kernel Operator Learning via Multilevel Trainingcited by 13
  2. 2023Monte Carlo Neural PDE Solver for Learning PDEs via Probabilistic Representationcited by 11
  3. 2025Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methodscited by 9
  4. 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximationcited by 4
  5. 2025Sobolev acceleration for neural networkscited by 3
  6. 2023Statistical Spatially Inhomogeneous Diffusion Inferencecited by 2
  7. 2025A Kernel-based Stochastic Approximation Framework for Nonlinear Operator Learningcited by 1
  8. 2023Number Theoretic Accelerated Learning of Physics-Informed Neural Networkscited by 0
  9. 2024Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Biascited by 0
  10. 2026Global Convergence of DGM and PINN Algorithms for Solving Nonlinear PDEscited by 0
  11. 2026Derivative-Informed Training of Neural Operators On-the-Fly via Sketched Tangent Consistencycited by 0