# Tame Riemannian Stochastic Approximation

arXiv: https://arxiv.org/abs/2302.00709 · HTML: https://mlsanity.com/paper/2302.00709/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 39 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 14 of 14)

- [Stochastic subgradient method converges on tame functions](https://mlsanity.com/paper/1804.07795/) (2018)
- [Geoopt: Riemannian Optimization in PyTorch](https://mlsanity.com/paper/2005.02819/) (2020)
- [Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning](https://mlsanity.com/paper/1909.10300/) (2019)
- [geomstats: a Python Package for Riemannian Geometry in Machine Learning](https://mlsanity.com/paper/1805.08308/) (2018)
- [A Brief Introduction to Manifold Optimization](https://mlsanity.com/paper/1906.05450/) (2019)
- [Convergence of constant step stochastic gradient descent for non-smooth non-convex functions](https://mlsanity.com/paper/2005.08513/) (2020)
- [Analysis of nonsmooth stochastic approximation: the differential inclusion approach](https://mlsanity.com/paper/1805.01916/) (2018)
- [On Riemannian Stochastic Approximation Schemes with Fixed Step-Size](https://mlsanity.com/paper/2102.07586/) (2021)
- [Stochastic Approximation on Riemannian manifolds](https://mlsanity.com/paper/1711.10754/) (2017)
- [Convergence Analysis of Riemannian Stochastic Approximation Schemes](https://mlsanity.com/paper/2005.13284/) (2020)
- [tensorflow-riemopt: A Library for Optimization on Riemannian Manifolds](https://mlsanity.com/paper/2105.13921/) (2021)
- [Retraction based Direct Search Methods for Derivative Free Riemannian Optimization](https://mlsanity.com/paper/2202.11052/) (2022)
- [Stochastic Langevin Differential Inclusions with Applications to Machine Learning](https://mlsanity.com/paper/2206.11533/) (2022)
- [Riemannian Stochastic Fixed Point Optimization Algorithm](https://mlsanity.com/paper/2012.09346/) (2020)

## Abstract (arXiv)

We study the properties of stochastic approximation applied to a tame nondifferentiable function subject to constraints defined by a Riemannian manifold. The objective landscape of tame functions, arising in o-minimal topology extended to a geometric category when generalized to manifolds, exhibits some structure that enables theoretical guarantees of expected function decrease and asymptotic convergence for generic stochastic sub-gradient descent. Recent work has shown that this class of functions faithfully model the loss landscape of deep neural network training objectives, and the autograd operation used in deep learning packages implements a variant of subgradient descent with the correct properties for convergence. Riemannian optimization uses geometric properties of a constraint set to perform a minimization procedure while enforcing adherence to the the optimization variable lying on a Riemannian manifold. This paper presents the first study of tame optimization on Riemannian manifolds, highlighting the rich geometric structure of the problem and confirming the appropriateness of the canonical "SGD" for such a problem with the analysis and numerical reports of a simple Retracted SGD algorithm.