# Natural Riemannian gradient for learning functional tensor networks

arXiv: https://arxiv.org/abs/2604.09263 · HTML: https://mlsanity.com/paper/2604.09263/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 39 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Learning Relevant Features of Data with Multi-scale Tensor Networks](https://mlsanity.com/paper/1801.00315/) (2017)
- [FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information](https://mlsanity.com/paper/2405.12807/) (2024)
- [Parallelized Tensor Train Learning of Polynomial Classifiers](https://mlsanity.com/paper/1612.06505/) (2016)
- [Gradient-based Optimization for Regression in the Functional Tensor-Train Format](https://mlsanity.com/paper/1801.00885/) (2018)
- [Tensor-based algorithms for image classification](https://mlsanity.com/paper/1910.02150/) (2019)
- [Riemannian Natural Gradient Methods](https://mlsanity.com/paper/2207.07287/) (2022)
- [Riemannian Optimization on Tree Tensor Networks with Application in Machine Learning](https://mlsanity.com/paper/2507.21726/) (2025)
- [Approximation and learning with compositional tensor trains](https://mlsanity.com/paper/2512.18059/) (2025)

## Built on by (most cited first, 1 of 1)

- [Stochastic Optimization of Tree Tensor Networks](https://mlsanity.com/paper/2609.00870/) (2026)

## Abstract (arXiv)

We consider machine learning tasks with low-rank functional tree tensor networks (TTN) as the learning model. While in the case of least-squares regression, low-rank functional TTNs can be efficiently optimized using alternating optimization, this is not directly possible in other problems, such as multinomial logistic regression. We propose a natural Riemannian gradient descent type approach applicable to arbitrary losses which is based on the natural gradient by Amari. In particular, the search direction obtained by the natural gradient is independent of the choice of basis of the underlying functional tensor product space. Our framework applies to both the factorized and manifold-based approach for representing the functional TTN. For practical application, we propose a hierarchy of efficient approximations to the true natural Riemannian gradient for computing the updates in the parameter space. Numerical experiments confirm our theoretical findings on common classification datasets and show that using natural Riemannian gradient descent for learning considerably improves convergence behavior when compared to standard Riemannian gradient methods.