# Riemannian Optimization in Modular Systems

arXiv: https://arxiv.org/abs/2603.03610 · HTML: https://mlsanity.com/paper/2603.03610/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 14 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 7 of 7)

- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [A Variational Perspective on Accelerated Methods in Optimization](https://mlsanity.com/paper/1603.04245/) (2016)
- [Scalable Optimization in the Modular Norm](https://mlsanity.com/paper/2405.14813/) (2024)
- [Natural Neural Networks](https://mlsanity.com/paper/1507.00210/) (2015)
- [Beyond Convexity -- Contraction and Global Convergence of Gradient Descent](https://mlsanity.com/paper/1806.06655/) (2018)
- [A singular Riemannian geometry approach to Deep Neural Networks II. Reconstruction of 1-D equivalence classes](https://mlsanity.com/paper/2112.10583/) (2021)
- [A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on n-dimensional Classes](https://mlsanity.com/paper/2404.06104/) (2024)

## Abstract (arXiv)

Understanding how systems built out of modular components can be jointly optimized is an important problem in biology, engineering, and machine learning. The backpropagation algorithm is one such solution and has been instrumental in the success of neural networks. Despite its empirical success, a strong theoretical understanding of it is lacking. Here, we combine tools from Riemannian geometry, optimal control theory, and theoretical physics to advance this understanding. We make three key contributions: First, we revisit the derivation of backpropagation as a constrained optimization problem and combine it with the insight that Riemannian gradient descent trajectories can be understood as the minimum of an action. Second, we introduce a recursively defined layerwise Riemannian metric that exploits the modular structure of neural networks and can be efficiently computed using the Woodbury matrix identity, avoiding the O(n3) cost of full metric inversion. Third, we develop a framework of composable ``Riemannian modules'' whose convergence properties can be quantified using nonlinear contraction theory, providing algorithmic stability guarantees of order O(kappa2 L/(xi μ √n)) where kappa and L are Lipschitz constants, μ is the mass matrix scale, and xi bounds the condition number. Our layerwise metric approach provides a practical alternative to natural gradient descent. While we focus here on studying neural networks, our approach more generally applies to the study of systems made of modules that are optimized over time, as it occurs in biology during both evolution and development.