mlsanityData to 14 Oct 2026

Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks

arXiv 2312.14499 ↗First posted 22 Dec 2023cs.LG cs.AI cs.NAcited by 23

Not on the map. Why?

Abstract (arXiv)

Physics-Informed Neural Networks (PINNs) have proven effective in solving partial differential equations (PDEs), especially when some data are available by seamlessly blending data and physics. However, extending PINNs to high-dimensional and even high-order PDEs encounters significant challenges due to the computational cost associated with automatic differentiation in the residual loss. Herein, we address the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE). Starting with the second-order high-dimensional PDEs ubiquitous in scientific computing, HTE transforms the calculation of the entire Hessian matrix into a Hessian vector product (HVP). This approach alleviates the computational bottleneck via Taylor-mode automatic differentiation and significantly reduces memory consumption from the Hessian matrix to HVP. We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions. Comparisons with Stochastic Dimension Gradient Descent (SDGD) highlight the distinct advantages of HTE, particularly in scenarios with significant variance among dimensions. We further extend HTE to higher-order and higher-dimensional PDEs, specifically addressing the biharmonic equation. By employing tensor-vector products (TVP), HTE efficiently computes the colossal tensor associated with the fourth-order high-dimensional biharmonic equation, saving memory and enabling rapid computation. The effectiveness of HTE is illustrated through experimental setups, demonstrating comparable convergence rates with SDGD under memory and speed constraints. Additionally, HTE proves valuable in accelerating the Gradient-Enhanced PINN (gPINN) version as well as the Biharmonic equation. Overall, HTE opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 32 of its 58 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

32 of 58 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 678
  5. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 583
  6. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  7. 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 473
  8. 2021Physics-informed neural networks with hard constraints for inverse designcited by 280
  9. 2016Deep Learning without Poor Local Minimacited by 234
  10. 2021Gradient-enhanced physics-informed neural networks for forward and inverse PDE problemscited by 222
  11. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 191
  12. 2018Adversarial Uncertainty Quantification in Physics-Informed Neural Networkscited by 120
20 more
  1. 2020Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEscited by 116
  2. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  3. 2023Tackling the Curse of Dimensionality with Physics-Informed Neural Networkscited by 79
  4. 2017Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equationscited by 65
  5. 2020Hutch++: Optimal Stochastic Trace Estimationcited by 63
  6. 2021When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?cited by 58
  7. 2018Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equationscited by 55
  8. 2019Deep splitting method for parabolic PDEscited by 53
  9. 2020Solving Inverse Stochastic Problems from Discrete Particle Observations Using the Fokker-Planck Equation and Physics-informed Neural Networkscited by 52
  10. 2021Meta-learning PINN loss functionscited by 51
  11. 2022Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodologycited by 44
  12. 2018Machine Learning for semi linear PDEscited by 28
  13. 2016Algorithms for Overcoming the Curse of Dimensionality for Certain Hamilton-Jacobi Equations Arising in Control Theory and Elsewherecited by 24
  14. 2020Randomized Automatic Differentiationcited by 23
  15. 2022Learning Physics-Informed Neural Networks without Stacked Back-propagationcited by 23
  16. 2019Solving high-dimensional optimal stopping problems using deep learningcited by 22
  17. 2019Overcoming the curse of dimensionality for some Hamilton--Jacobi partial differential equations via neural network architecturescited by 22
  18. 2023Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEscited by 19
  19. 2019Three algorithms for solving high-dimensional fully-coupled FBSDEs through deep learningcited by 15
  20. 2020A Modern Analysis of Hutchinson's Trace Estimatorcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 7; 2026: 8.

23 papers.

Sort:

  1. 2024Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck Equationscited by 22
  2. 2024Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operatorscited by 20
  3. 2024PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representationscited by 11
  4. 2024Separable Physics-informed Neural Networks for Solving the BGK Model of the Boltzmann Equationcited by 8
  5. 2024Spectral Informed Neural Network: An Efficient and Low-Memory PINNcited by 6
  6. 2024DOF: Accelerating High-order Differential Operators with Forward Propagationcited by 5
  7. 2024Tackling the Curse of Dimensionality in Fractional and Tempered Fractional PDEs with Physics-Informed Neural Networkscited by 5
  8. 2025A brief review of the Deep BSDE method for solving high-dimensional partial differential equationscited by 5
  9. 2025Integration Matters for Learning PDEs with Backward SDEscited by 3
  10. 2025Noisy PDE Training Requires Bigger PINNscited by 2
  11. 2026Physics-informed neural particle flow for the Bayesian update stepcited by 2
  12. 2024Score-fPINN: Fractional Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck-Levy Equationscited by 1
11 more
  1. 2025Collapsing Taylor Mode Automatic Differentiationcited by 1
  2. 2025Tensor-Compressed and Fully-Quantized Training of Neural PDE Solverscited by 1
  3. 2026Deep Neural networks for solving high-dimensional parabolic partial differential equationscited by 1
  4. 2025Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifoldscited by 0
  5. 2025Breaking the Precision Ceiling in Physics-Informed Neural Networks: A Hybrid Fourier-Neural Architecture for Ultra-High Accuracycited by 0
  6. 2026Jacobian Regularization Stabilizes Long-Term Integration of Neural Differential Equationscited by 0
  7. 2026Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNscited by 0
  8. 2026Stochastic Dimension Implicit Functional Projections for Global Integral Conservation in High-Dimensional PINNscited by 0
  9. 2026Stochastic-Dimension Frozen Sampled Neural Network for High-Dimensional Gross-Pitaevskii Equations on Unbounded Domainscited by 0
  10. 2026Unbiased and Second-Order-Free Training for High-Dimensional PDEscited by 0
  11. 2026Parameterized Representations via Implicit Stochastic Modulation for High-Dimensional and High-Order Neural PDE Solverscited by 0