mlsanityData to 14 Oct 2026

Sobolev acceleration for neural networks

arXiv 2509.19773 ↗First posted 24 Sep 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Sobolev training, which integrates target derivatives into the loss functions, has been shown to accelerate convergence and improve generalization compared to conventional L2 training. However, the underlying mechanisms of this training method remain only partially understood. In this work, we present the first rigorous theoretical framework proving that Sobolev training accelerates the convergence of Rectified Linear Unit (ReLU) networks. Under a student-teacher framework with Gaussian inputs and shallow architectures, we derive exact formulas for population gradients and Hessians, and quantify the improvements in conditioning of the loss landscape and gradient-flow convergence rates. Extensive numerical experiments validate our theoretical findings and show that the benefits of Sobolev training extend to modern deep learning tasks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

36 of 51 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  9. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  10. 2018On the Spectral Bias of Neural Networkscited by 1,341
  11. 2016An overview of gradient descent optimization algorithmscited by 1,033
  12. 2022Diffusion Models in Vision: A Surveycited by 1,015
24 more
  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  3. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  4. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 583
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  6. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  9. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 284
  10. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  11. 2017Sobolev Training for Neural Networkscited by 143
  12. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  13. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 113
  14. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  15. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  16. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  17. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  18. 2022Derivative-Informed Neural Operator: An Efficient Framework for High-Dimensional Parametric Derivative Learningcited by 44
  19. 2019Multi-scale Deep Neural Networks for Solving High Dimensional PDEscited by 25
  20. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  21. 2022Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descentcited by 11
  22. 2019Learning Distributions Generated by One-Layer ReLU Networkscited by 10
  23. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
  24. 2020Global Convergence of Sobolev Training for Overparameterized Neural Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Neu-PiG: Neural Preconditioned Grids for Fast Dynamic Surface Reconstruction on Long Sequencescited by 1
  2. 2026LEIA: Learned Environment for Interactive Architected Materialscited by 0
  3. 2026Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networkscited by 0