Zeroth-order Riemannian Averaging Stochastic Approximation Algorithms

We present Zeroth-order Riemannian Averaging Stochastic Approximation (textttZo-RASA) algorithms for stochastic optimization on Riemannian manifolds. We show that textttZo-RASA achieves optimal sample complexities for generating ε-approximation first-order stationary solutions using only one-sample or constant-order batches in each iteration. Our approach employs Riemannian moving-average stochastic gradient estimators, and a novel Riemannian-Lyapunov analysis technique for convergence analysis. We improve the algorithm's practicality by using retractions and vector transport, instead of exponential mappings and parallel transports, thereby reducing per-iteration complexity. Additionally, we introduce a novel geometric condition, satisfied by manifolds with bounded second fundamental form, which enables new error bounds for approximating parallel transport with vector transport.