A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks
Not on the map. Why?
Abstract (arXiv)
The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose n-TangentProp, the natural extension of the TangentProp formalism citesimard1991tangent to arbitrarily many derivatives. n-TangentProp computes the exact derivative dn/dxn f(x) in quasilinear, instead of exponential time, for a densely connected, feed-forward neural network f with a smooth, parameter-free activation function. We validate our algorithm empirically across a range of depths, widths, and number of derivatives. We demonstrate that our method is particularly beneficial in the context of physics-informed neural networks where ntp allows for significantly faster training times than previous methods and has favorable scaling with respect to both model size and loss-function complexity as measured by the number of required derivatives. The code for this paper can be found at https://github.com/kyrochi/ntangentprop.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 12 of its 42 references that are in the map. First posted on arXiv 12 Dec 2024.
Builds on
12 of 42 references.
References · 12
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018On the Spectral Bias of Neural Networkscited by 1,321
- 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
- 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
- 2020Understanding and mitigating gradient pathologies in physics-informed neural networkscited by 134
- 2023An Expert's Guide to Training Physics-informed Neural Networkscited by 119
- 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 109
- 2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 86
- 2023An operator preconditioning perspective on training in physics-informed machine learningcited by 27
- 2024Recent Advances on Machine Learning for Computational Fluid Dynamics: A Surveycited by 27
- 2022Accelerated Training of Physics-Informed Neural Networks (PINNs) using Meshless Discretizationscited by 26
- 2024Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Losscited by 14
Built on by
Citing papers per year: 2026: 2.
2 papers.