# Automatic Differentiation from Scratch: How PyTorch Computes Gradients in Physics-Informed Neural Networks

arXiv: https://arxiv.org/abs/2607.13042 · HTML: https://mlsanity.com/paper/2607.13042/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 13 Jun 2026.

## Builds on (most cited first, 7 of 7)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [An Expert's Guide to Training Physics-informed Neural Networks](https://mlsanity.com/paper/2308.08468/) (2023)
- [Physics-Informed Neural Networks: A Didactic Derivation of the Complete Training Cycle](https://mlsanity.com/paper/2604.18481/) (2026)

## Abstract (arXiv)

This paper traces, with explicit numerical values, how PyTorch's automatic differentiation (AD) engine computes gradients for Physics-Informed Neural Network (PINN) training -- a setting that requires two levels of differentiation: computing the physics derivative haty'(t)=dhaty/dt through the network, and computing parameter gradients nablaθ L of a loss that itself depends on haty'(t). Using a 1-3-3-1 multilayer perceptron and the initial value problem y'(t)+y(t)=0, y(0)=1, we trace the complete pipeline at every node: the computational graph built during the forward pass, the reverse-mode backward traversal that computes all 22 parameter gradients in a single pass, and the graph-on-graph mechanism by which textttcreategraph=True enables correct differentiation through the physics-informed residual. Every adjoint value is verified against the hand derivations of Tahimi (2026), connecting the P/Q sensitivity framework to the vector--Jacobian products used by PyTorch's autograd engine.