# Riemannian Adaptive Optimization Algorithm and Its Application to Natural Language Processing

arXiv: https://arxiv.org/abs/2004.00897 · HTML: https://mlsanity.com/paper/2004.00897/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2020.

## Built on by (most cited first, 7 of 7)

- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2303.00565/) (2023)
- [Vision Transformer with Convolutions Architecture Search](https://mlsanity.com/paper/2203.10435/) (2022)
- [Adaptive Aggregation with Two Gains in QFL](https://mlsanity.com/paper/2512.03363/) (2025)
- [G-Mix: A Generalized Mixup Learning Framework Towards Flat Minima](https://mlsanity.com/paper/2308.03236/) (2023)
- [Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks](https://mlsanity.com/paper/2609.21039/) (2026)
- [Temporal Geometry of Deep Networks: Hyperbolic Representations of Training Dynamics for Intrinsic Explainability](https://mlsanity.com/paper/2610.03000/) (2026)

## Abstract (arXiv)

This paper proposes a Riemannian adaptive optimization algorithm to optimize the parameters of deep neural networks. The algorithm is an extension of both AMSGrad in Euclidean space and RAMSGrad on a Riemannian manifold. The algorithm helps to resolve two issues affecting RAMSGrad. The first is that it can solve the Riemannian stochastic optimization problem directly, in contrast to RAMSGrad which only achieves a low regret. The other is that it can use constant learning rates, which makes it implementable in practice. Additionally, we apply the proposed algorithm to Poincar'e embeddings, which embed the transitive closure of the WordNet nouns into the Poincar'e ball model of hyperbolic space. Numerical experiments show that regardless of the initial value of the learning rate, our algorithm stably converges to the optimal solution and converges faster than RSGD, the most basic Riemannian stochastic optimization algorithm.