# Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds

arXiv: https://arxiv.org/abs/2103.14974 · HTML: https://mlsanity.com/paper/2103.14974/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 34 references that are in the map.
- First posted on arXiv 27 Mar 2021.

## Builds on (most cited first, 5 of 5)

- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [TensorLy: Tensor Learning in Python](https://mlsanity.com/paper/1610.09555/) (2016)
- [Pymanopt: A Python Toolbox for Optimization on Manifolds using Automatic Differentiation](https://mlsanity.com/paper/1603.03236/) (2016)
- [Manifold Geometry with Fast Automatic Derivatives and Coordinate Frame Semantics Checking in C++](https://mlsanity.com/paper/1805.01810/) (2018)
- [Second-order optimization for tensors with fixed tensor-train rank](https://mlsanity.com/paper/2011.13395/) (2020)

## Built on by (most cited first, 6 of 6)

- [TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning](https://mlsanity.com/paper/2205.00293/) (2022)
- [LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters](https://mlsanity.com/paper/2507.12142/) (2025)
- [Quantized Fourier and Polynomial Features for more Expressive Tensor Network Models](https://mlsanity.com/paper/2309.05436/) (2023)
- [Tensor-Train Point Cloud Compression and Efficient Approximate Nearest-Neighbor Search](https://mlsanity.com/paper/2410.04462/) (2024)
- [Higher Order Reduced Rank Regression](https://mlsanity.com/paper/2503.06528/) (2025)
- [Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices](https://mlsanity.com/paper/2609.27982/) (2026)

## Abstract (arXiv)

In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding low-rank approximations is to use Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this paper, we build upon automatic differentiation and propose a method that, given an implementation of the function to be minimized, efficiently computes Riemannian gradients and matrix-by-vector products between an approximate Riemannian Hessian and a given vector.