# Position: Optimization in SciML Should Employ the Function Space Geometry

arXiv: https://arxiv.org/abs/2402.07318 · HTML: https://mlsanity.com/paper/2402.07318/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2024.

## Built on by (most cited first, 20 of 20)

- [Lie Algebra Canonicalization: Equivariant Neural Operators under arbitrary Lie Groups](https://mlsanity.com/paper/2410.02698/) (2024)
- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Dual Cone Gradient Descent for Training Physics-Informed Neural Networks](https://mlsanity.com/paper/2409.18426/) (2024)
- [Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization](https://mlsanity.com/paper/2505.12149/) (2025)
- [Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning](https://mlsanity.com/paper/2506.10973/) (2025)
- [Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks](https://mlsanity.com/paper/2604.05230/) (2026)
- [ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning](https://mlsanity.com/paper/2412.10782/) (2024)
- [Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.21404/) (2025)
- [Functional Neural Wavefunction Optimization](https://mlsanity.com/paper/2507.10835/) (2025)
- [Generalized Lie Symmetries in Physics-Informed Neural Operators](https://mlsanity.com/paper/2502.00373/) (2025)
- [Riemann Tensor Neural Networks: Learning Conservative Systems with Physics-Constrained Networks](https://mlsanity.com/paper/2503.00755/) (2025)
- [A Sketch-and-Project Analysis of Subsampled Natural Gradient Algorithms](https://mlsanity.com/paper/2508.21022/) (2025)
- [Natural gradient descent with momentum](https://mlsanity.com/paper/2604.15554/) (2026)
- [Accelerating Natural Gradient Descent for PINNs with Randomized Numerical Linear Algebra](https://mlsanity.com/paper/2505.11638/) (2025)
- [Can Data-Driven Dynamics Reveal Hidden Physics? There Is A Need for Interpretable Neural Operators](https://mlsanity.com/paper/2510.02683/) (2025)
- [Approximation and learning with compositional tensor trains](https://mlsanity.com/paper/2512.18059/) (2025)
- [Random Gradient-Free Optimization in Infinite Dimensional Spaces](https://mlsanity.com/paper/2512.20566/) (2025)
- [Preconditioned Physics-Informed Neural Operator Training](https://mlsanity.com/paper/2609.36216/) (2026)
- [Gauss-Newton Natural Gradient Descent for Shape Learning](https://mlsanity.com/paper/2602.00099/) (2026)
- [Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers](https://mlsanity.com/paper/2607.22004/) (2026)

## Abstract (arXiv)

Scientific machine learning (SciML) is a relatively new field that aims to solve problems from different fields of natural sciences using machine learning tools. It is well-documented that the optimizers commonly used in other areas of machine learning perform poorly on many SciML problems. We provide an infinite-dimensional view on optimization problems encountered in scientific machine learning and advocate for the paradigm first optimize, then discretize for their solution. This amounts to first choosing an appropriate infinite-dimensional algorithm which is then discretized in a second step. To illustrate this point, we show that recently proposed state-of-the-art algorithms for SciML applications can be derived within this framework. As the infinite-dimensional viewpoint is presently underdeveloped in scientific machine learning, we formalize it here and advocate for its use in SciML in the development of efficient optimization algorithms.