# Deep Relaxation: partial differential equations for optimizing deep neural networks

arXiv: https://arxiv.org/abs/1704.04932 · HTML: https://mlsanity.com/paper/1704.04932/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2017.

## Built on by (most cited first, 27 of 27)

- [PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/2108.01938/) (2021)
- [RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control](https://mlsanity.com/paper/2405.17401/) (2024)
- [SAM as an Optimal Relaxation of Bayes](https://mlsanity.com/paper/2210.01620/) (2022)
- [Stochastic Optimal Control Matching](https://mlsanity.com/paper/2312.02027/) (2023)
- [Stochastic Optimal Control for Diffusion Bridges in Function Spaces](https://mlsanity.com/paper/2405.20630/) (2024)
- [Value Gradient Guidance for Flow Matching Alignment](https://mlsanity.com/paper/2512.05116/) (2025)
- [A Taxonomy of Loss Functions for Stochastic Optimal Control](https://mlsanity.com/paper/2410.00345/) (2024)
- [On the Hyperparameters in Stochastic Gradient Descent with Momentum](https://mlsanity.com/paper/2108.03947/) (2021)
- [Ensemble Inference Methods for Models With Noisy and Expensive Likelihoods](https://mlsanity.com/paper/2104.03384/) (2021)
- [Learning Rate Schedules in the Presence of Distribution Shift](https://mlsanity.com/paper/2303.15634/) (2023)
- [Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs](https://mlsanity.com/paper/2204.08621/) (2022)
- [On Large Batch Training and Sharp Minima: A Fokker-Planck Perspective](https://mlsanity.com/paper/2112.00987/) (2021)
- [dNNsolve: an efficient NN-based PDE solver](https://mlsanity.com/paper/2103.08662/) (2021)
- [Adversarial Training for Gradient Descent: Analysis Through its Continuous-time Approximation](https://mlsanity.com/paper/2105.08037/) (2021)
- [Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations](https://mlsanity.com/paper/2109.00095/) (2021)
- [Enhance Diffusion to Improve Robust Generalization](https://mlsanity.com/paper/2306.02618/) (2023)
- [Multigoal-oriented dual-weighted-residual error estimation using deep neural networks](https://mlsanity.com/paper/2112.11360/) (2021)
- [HJ-sampler: A Bayesian sampler for inverse problems of a stochastic process by leveraging Hamilton-Jacobi PDEs and score-based generative models](https://mlsanity.com/paper/2409.09614/) (2024)
- [Time-Frequency Analysis for Neural Networks](https://mlsanity.com/paper/2512.15992/) (2025)
- [Cell-average based neural network method for hyperbolic and parabolic partial differential equations](https://mlsanity.com/paper/2107.00813/) (2021)
- [A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks](https://mlsanity.com/paper/2206.02001/) (2022)
- [Residual networks classify inputs based on their neural transient dynamics](https://mlsanity.com/paper/2101.03009/) (2021)
- [Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations](https://mlsanity.com/paper/2101.03583/) (2021)
- [Error Propagation in Dynamic Programming: From Stochastic Control to American Option Pricing](https://mlsanity.com/paper/2509.20239/) (2025)
- [Deep learning methods for inverse problems using connections between proximal operators and Hamilton-Jacobi equations](https://mlsanity.com/paper/2512.23829/) (2025)
- [A Mean Field Games Perspective on Evolutionary Clustering](https://mlsanity.com/paper/2603.27137/) (2026)
- [A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks](https://mlsanity.com/paper/2606.18303/) (2026)

## Abstract (arXiv)

In this paper we establish a connection between non-convex optimization methods for training deep neural networks and nonlinear partial differential equations (PDEs). Relaxation techniques arising in statistical physics which have already been used successfully in this context are reinterpreted as solutions of a viscous Hamilton-Jacobi PDE. Using a stochastic control interpretation allows we prove that the modified algorithm performs better in expectation that stochastic gradient descent. Well-known PDE regularity results allow us to analyze the geometry of the relaxed energy landscape, confirming empirical evidence. The PDE is derived from a stochastic homogenization problem, which arises in the implementation of the algorithm. The algorithms scale well in practice and can effectively tackle the high dimensionality of modern neural networks.