mlsanityData to 7 Oct 2026

Stochastic Dimension Zeroth-Order Estimator: Stable and Memory-Efficient Training of PINNs

arXiv 2603.24002 ↗First posted 25 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Physics-Informed Neural Networks (PINNs) for high-dimensional and high-order partial differential equations (PDEs) are primarily constrained by the O(dk) spatial derivative complexity and the O(P) memory overhead of backpropagation (BP). While randomized spatial estimators successfully reduce the spatial complexity to O(1), their reliance on first-order optimization still leads to prohibitive memory consumption at scale. Zeroth-order (ZO) optimization offers a BP-free alternative; however, naively combining randomized spatial operators with ZO perturbations triggers a variance explosion of O(1/varepsilon2), leading to numerical divergence. To address these challenges, we propose the Stochastic Dimension-free Zeroth-order Estimator (SDZE), a unified framework that achieves dimension-independent complexity in both space and memory. Specifically, SDZE leverages Common Random Numbers Synchronization (CRNS) to algebraically cancel the O(1/varepsilon2) variance by locking spatial random seeds across perturbations. Furthermore, an implicit matrix-free subspace projection is introduced to reduce parameter exploration variance from O(P) to O(r) while maintaining an O(1) optimizer memory footprint. Empirical results demonstrate that SDZE enables the training of 10-million-dimensional PINNs on a single NVIDIA A100 GPU, delivering significant improvements in speed and memory efficiency over state-of-the-art baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

41 of 65 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  3. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
  4. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  5. 2021Physics-informed neural networks with hard constraints for inverse designcited by 277
  6. 2021Gradient-enhanced physics-informed neural networks for forward and inverse PDE problemscited by 220
  7. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  8. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 160
  9. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  10. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  11. 2021CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Methodcited by 93
  12. 2022Tutorial on amortized optimizationcited by 83
29 more
  1. 2023Tackling the Curse of Dimensionality with Physics-Informed Neural Networkscited by 78
  2. 2022Gradients without Backpropagationcited by 73
  3. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 66
  4. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  5. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  6. 2019Improved Zeroth-Order Variance Reduced Algorithms and Analysis for Nonconvex Optimizationcited by 47
  7. 2020Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Samplingcited by 45
  8. 2023Randomized Numerical Linear Algebra : A Perspective on the Field With an Eye to Softwarecited by 38
  9. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  10. 2023ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimizationcited by 24
  11. 2020Randomized Automatic Differentiationcited by 23
  12. 2023Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networkscited by 23
  13. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  14. 2024Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck Equationscited by 22
  15. 2021Zeroth order optimization with orthogonal random directionscited by 21
  16. 2024Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operatorscited by 20
  17. 2020Unbiased MLMC stochastic gradient-based optimization of Bayesian experimental designscited by 19
  18. 2023Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEscited by 19
  19. 2020A stochastic subspace approach to gradient-free optimization in high dimensionscited by 15
  20. 2022Stochastic projection based approach for gradient free physics informed learningcited by 15
  21. 2023Zeroth-order Optimization with Weak Dimension Dependencycited by 11
  22. 2025Anant-Net: Breaking the Curse of Dimensionality with Scalable and Interpretable Neural Surrogate for High-Dimensional PDEscited by 11
  23. 2019Accelerating Variance-Reduced Stochastic Gradient Methodscited by 10
  24. 2021Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimizationcited by 10
  25. 2021Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Surveycited by 8
  26. 2022Direct search based on probabilistic descent in reduced spacescited by 7
  27. 2024Zeroth-order Random Subspace Algorithm for Non-smooth Convex Optimizationcited by 6
  28. 2024DOF: Accelerating High-order Differential Operators with Forward Propagationcited by 5
  29. 2024Tackling the Curse of Dimensionality in Fractional and Tempered Fractional PDEs with Physics-Informed Neural Networkscited by 5

Built on by

Not cited yet.