# Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates

arXiv: https://arxiv.org/abs/2607.07623 · HTML: https://mlsanity.com/paper/2607.07623/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 8 Jul 2026.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Old Optimizer, New Norm: An Anthology](https://mlsanity.com/paper/2409.20325/) (2024)
- [Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients](https://mlsanity.com/paper/1705.07774/) (2017)
- [Challenges in Training PINNs: A Loss Landscape Perspective](https://mlsanity.com/paper/2402.01868/) (2024)
- [Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling](https://mlsanity.com/paper/2207.02338/) (2022)
- [The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.03864/) (2024)
- [Accelerating Natural Gradient with Higher-Order Invariance](https://mlsanity.com/paper/1803.01273/) (2018)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Ensemble learning for Physics Informed Neural Networks: a Gradient Boosting approach](https://mlsanity.com/paper/2302.13143/) (2023)
- [Learning a Neural Solver for Parametric PDE to Enhance Physics-Informed Methods](https://mlsanity.com/paper/2410.06820/) (2024)
- [A Coordinate-Free Construction of Scalable Natural Gradient](https://mlsanity.com/paper/1808.10340/) (2018)

## Abstract (arXiv)

Nonlinear least-squares objectives form the foundation of scientific machine-learning tasks, yet even curvature-aware optimizers remain geometrically inconsistent at finite step sizes: Levenberg-Marquardt (LM) derives its direction from local Riemannian metrics but realizes it as a straight parameter update. Here we introduce RNC-LM, which carries the LM direction along a locally constructed curved trajectory in Riemann normal coordinates. A recursive reformulation of the geodesic equation generates arbitrary finite-order corrections while reusing the same damped Gauss-Newton matrix factorization, and curve length is controlled separately from damping. On a reaction-diffusion physics-informed neural network benchmark, RNC-LM reduces relative L2 errors below 8×10-3, whereas L-BFGS, LM and LM with geodesic acceleration remain near one. On a large-scale machine-learning potential fitting task with 985,160 configurations, fourth-order RNC-LM reaches a fixed training-error target with a 34× wall-clock speedup over LM. These results establish finite-step geometric realization as a distinct optimizer design principle.