# Tangent-Space Gradient Optimization of Tensor Network for Machine Learning

arXiv: https://arxiv.org/abs/2001.04029 · HTML: https://mlsanity.com/paper/2001.04029/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2020.

## Built on by (most cited first, 7 of 7)

- [Tensor networks for interpretable and efficient quantum-inspired machine learning](https://mlsanity.com/paper/2311.11258/) (2023)
- [Improvements to Gradient Descent Methods for Quantum Tensor Network Machine Learning](https://mlsanity.com/paper/2203.03366/) (2022)
- [tn4ml: Tensor Network Training and Customization for Machine Learning](https://mlsanity.com/paper/2502.13090/) (2025)
- [How to Square Tensor Networks and Circuits Without Squaring Them](https://mlsanity.com/paper/2512.17090/) (2025)
- [Tensor network to learn the wavefunction of data](https://mlsanity.com/paper/2111.08014/) (2021)
- [Non-parametric Semi-Supervised Learning in Many-body Hilbert Space with Rescaled Logarithmic Fidelity](https://mlsanity.com/paper/2107.00195/) (2021)
- [Gaussian Universality and Its Breakdown in Tensor-Network Machine Learning](https://mlsanity.com/paper/2610.06080/) (2026)

## Abstract (arXiv)

The gradient-based optimization method for deep machine learning models suffers from gradient vanishing and exploding problems, particularly when the computational graph becomes deep. In this work, we propose the tangent-space gradient optimization (TSGO) for the probabilistic models to keep the gradients from vanishing or exploding. The central idea is to guarantee the orthogonality between the variational parameters and the gradients. The optimization is then implemented by rotating parameter vector towards the direction of gradient. We explain and testify TSGO in tensor network (TN) machine learning, where the TN describes the joint probability distribution as a normalized state left| psi rightrangle  in Hilbert space. We show that the gradient can be restricted in the tangent space of leftlangle psi right.left| psi rightrangle = 1 hyper-sphere. Instead of additional adaptive methods to control the learning rate in deep learning, the learning rate of TSGO is naturally determined by the angle θ  as η = tan θ . Our numerical results reveal better convergence of TSGO in comparison to the off-the-shelf Adam.