# The Hamilton-Jacobi Theory of Deep Learning

arXiv: https://arxiv.org/abs/2605.28983 · HTML: https://mlsanity.com/paper/2605.28983/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 66 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 14 of 14)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [A Neural Scaling Law from the Dimension of the Data Manifold](https://mlsanity.com/paper/2004.10802/) (2020)
- [Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape](https://mlsanity.com/paper/2402.01258/) (2024)
- [Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective](https://mlsanity.com/paper/1908.10920/) (2019)
- [Overcoming the curse of dimensionality for some Hamilton--Jacobi partial differential equations via neural network architectures](https://mlsanity.com/paper/1910.09045/) (2019)
- [Scaling Laws for Linear Complexity Language Models](https://mlsanity.com/paper/2406.16690/) (2024)
- [Neural Networks are Decision Trees](https://mlsanity.com/paper/2210.05189/) (2022)
- [Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning](https://mlsanity.com/paper/2503.01329/) (2025)
- [SETOL: A Semi-Empirical Theory of (Deep) Learning](https://mlsanity.com/paper/2507.17912/) (2025)
- [How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNs](https://mlsanity.com/paper/2602.05779/) (2026)

## Abstract (arXiv)

In this paper, training a neural network is identified, exactly, as a search through Hamilton--Jacobi initial-value problems: each gradient step selects the initial data of a viscous Hamilton--Jacobi equation whose Hopf--Cole propagator best fits the observations; at inference, the input is the spatial point at which that solution is evaluated and the initial condition is already encoded in the weights. The correspondence is exact for log-sum-exp layers, with ReLU, sigmoid, SiLU, and GELU each an exact limit, gradient, or moment of the same object, and exact in composition across depth and width, with a quantified error at finite depth that vanishes in the joint limit. It is structural for residual networks, transformers, and recurrent networks (RNNs, LSTMs, SSMs), each discretizing the same class of equations, at a named and quantified approximation error. A single deformation parameter varepsilon unifies all four perspectives (network, tropical algebra, viscous PDE, convex optimization) in a commutative diagram closed under Lipschitz conditions. Quantitative consequences include: the minimax optimal generalization rate O(n-1/(d+2)) for fixed t; adversarial robustness controlled by varepsilon; backpropagation as the co-state equation of the Hamiltonian system for residual networks (Pontryagin Maximum Principle); scaling exponents consistent with data intrinsic dimension via PDE quadrature; and a closed-form O(N) influence function (softmax attribution weights πj) whose entropy landscape undergoes fold bifurcations as varepsilon increases, each merging attribution basins.