mlsanityData to 14 Oct 2026

Deep Relaxation: partial differential equations for optimizing deep neural networks

arXiv 1704.04932 ↗First posted 17 Apr 2017cs.LG math.AP math.OCcited by 27

Not on the map. Why?

Abstract (arXiv)

In this paper we establish a connection between non-convex optimization methods for training deep neural networks and nonlinear partial differential equations (PDEs). Relaxation techniques arising in statistical physics which have already been used successfully in this context are reinterpreted as solutions of a viscous Hamilton-Jacobi PDE. Using a stochastic control interpretation allows we prove that the modified algorithm performs better in expectation that stochastic gradient descent. Well-known PDE regularity results allow us to analyze the geometry of the relaxed energy landscape, confirming empirical evidence. The PDE is derived from a stochastic homogenization problem, which arises in the implementation of the algorithm. The algorithms scale well in practice and can effectively tackle the high dimensionality of modern neural networks.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 3; 2023: 3; 2024: 4; 2025: 4; 2026: 2.

27 papers.

Sort:

By area:

  1. 2021PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equationscited by 106
  2. 2024RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Controlcited by 48
  3. 2022SAM as an Optimal Relaxation of Bayescited by 41
  4. 2023Stochastic Optimal Control Matchingcited by 40
  5. 2024Stochastic Optimal Control for Diffusion Bridges in Function Spacescited by 16
  6. 2025Value Gradient Guidance for Flow Matching Alignmentcited by 14
  7. 2024A Taxonomy of Loss Functions for Stochastic Optimal Controlcited by 13
  8. 2021On the Hyperparameters in Stochastic Gradient Descent with Momentumcited by 9
  9. 2021Ensemble Inference Methods for Models With Noisy and Expensive Likelihoodscited by 8
  10. 2023Learning Rate Schedules in the Presence of Distribution Shiftcited by 7
  11. 2022Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEscited by 6
  12. 2021On Large Batch Training and Sharp Minima: A Fokker-Planck Perspectivecited by 5
15 more
  1. 2021dNNsolve: an efficient NN-based PDE solvercited by 4
  2. 2021Adversarial Training for Gradient Descent: Analysis Through its Continuous-time Approximationcited by 4
  3. 2021Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equationscited by 3
  4. 2023Enhance Diffusion to Improve Robust Generalizationcited by 3
  5. 2021Multigoal-oriented dual-weighted-residual error estimation using deep neural networkscited by 2
  6. 2024HJ-sampler: A Bayesian sampler for inverse problems of a stochastic process by leveraging Hamilton-Jacobi PDEs and score-based generative modelscited by 2
  7. 2025Time-Frequency Analysis for Neural Networkscited by 2
  8. 2021Cell-average based neural network method for hyperbolic and parabolic partial differential equationscited by 1
  9. 2022A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networkscited by 1
  10. 2021Residual networks classify inputs based on their neural transient dynamicscited by 0
  11. 2021Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equationscited by 0
  12. 2025Error Propagation in Dynamic Programming: From Stochastic Control to American Option Pricingcited by 0
  13. 2025Deep learning methods for inverse problems using connections between proximal operators and Hamilton-Jacobi equationscited by 0
  14. 2026A Mean Field Games Perspective on Evolutionary Clusteringcited by 0
  15. 2026A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networkscited by 0