# Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations

arXiv: https://arxiv.org/abs/2405.14099 · HTML: https://mlsanity.com/paper/2405.14099/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 44 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain](https://mlsanity.com/paper/2004.13145/) (2020)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method](https://mlsanity.com/paper/2110.15832/) (2021)
- [Error analysis for physics informed neural networks (PINNs) approximating Kolmogorov PDEs](https://mlsanity.com/paper/2106.14473/) (2021)
- [Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives](https://mlsanity.com/paper/2305.08466/) (2023)
- [Teaching the Incompressible Navier-Stokes Equations to Fast Neural Surrogate Models in 3D](https://mlsanity.com/paper/2012.11893/) (2020)
- [Transformer Meets Boundary Value Inverse Problems](https://mlsanity.com/paper/2209.14977/) (2022)
- [Finite Volume Neural Network: Modeling Subsurface Contaminant Transport](https://mlsanity.com/paper/2104.06010/) (2021)
- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [DOSnet as a Non-Black-Box PDE Solver: When Deep Learning Meets Operator Splitting](https://mlsanity.com/paper/2212.05571/) (2022)
- [Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2309.15244/) (2023)

## Built on by (most cited first, 9 of 9)

- [Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers](https://mlsanity.com/paper/2410.06308/) (2024)
- [Learn Singularly Perturbed Solutions via Homotopy Dynamics](https://mlsanity.com/paper/2502.00488/) (2025)
- [DeepONet for Solving Nonlinear Partial Differential Equations with Physics-Informed Training](https://mlsanity.com/paper/2410.04344/) (2024)
- [ZENN: A Thermodynamics-Inspired Computational Framework for Heterogeneous Data-Driven Modeling](https://mlsanity.com/paper/2505.09851/) (2025)
- [ARDO: A Weak Formulation Deep Neural Network Method for Elliptic and Parabolic PDEs Based on Random Differences of Test Functions](https://mlsanity.com/paper/2509.03757/) (2025)
- [A Neural Network for the Identical Kuramoto Equation: Architectural Considerations and Performance Evaluation](https://mlsanity.com/paper/2509.14384/) (2025)
- [Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Parameter Compression of Deep Neural Networks](https://mlsanity.com/paper/2606.00130/) (2026)
- [Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond](https://mlsanity.com/paper/2608.11020/) (2026)
- [A Systematic Analysis of Automatic Differentiation versus Discretization-based Constraints for Physics-Informed PDE Solvers](https://mlsanity.com/paper/2609.07437/) (2026)

## Abstract (arXiv)

Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering, especially in scenarios featuring complex domains or incorporation of empirical data. One advantage of the neural network methods for PDEs lies in its automatic differentiation (AD), which necessitates only the sample points themselves, unlike traditional finite difference (FD) approximations that require nearby local points to compute derivatives. In this paper, we quantitatively demonstrate the advantage of AD in training neural networks. The concept of truncated entropy is introduced to characterize the training property. Specifically, through comprehensive experimental and theoretical analyses conducted on random feature models and two-layer neural networks, we discover that the defined truncated entropy serves as a reliable metric for quantifying the residual loss of random feature models and the training speed of neural networks for both AD and FD methods. Our experimental and theoretical analyses demonstrate that, from a training perspective, AD outperforms FD in solving PDEs.