# A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints

arXiv: https://arxiv.org/abs/2208.13643 · HTML: https://mlsanity.com/paper/2208.13643/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 29 Aug 2022.

## Builds on (most cited first, 19 of 19)

- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization](https://mlsanity.com/paper/1709.08765/) (2017)
- [Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks](https://mlsanity.com/paper/1709.06079/) (2017)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds](https://mlsanity.com/paper/1605.07147/) (2016)
- [On Nonconvex Decentralized Gradient Descent](https://mlsanity.com/paper/1608.05766/) (2016)
- [A Brief Introduction to Manifold Optimization](https://mlsanity.com/paper/1906.05450/) (2019)
- [Decentralized Riemannian Gradient Descent on the Stiefel Manifold](https://mlsanity.com/paper/2102.07091/) (2021)
- [A general framework for decentralized optimization with first-order methods](https://mlsanity.com/paper/2009.05837/) (2020)
- [Multi-Agent Distributed Optimization via Inexact Consensus ADMM](https://mlsanity.com/paper/1402.6065/) (2014)
- [Fast decentralized non-convex finite-sum optimization with recursive variance reduction](https://mlsanity.com/paper/2008.07428/) (2020)
- [Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data](https://mlsanity.com/paper/1412.7839/) (2014)
- [Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints](https://mlsanity.com/paper/1810.03930/) (2018)
- [Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach](https://mlsanity.com/paper/1910.05857/) (2019)
- [Multipliers Correction Methods for Optimization Problems over the Stiefel Manifold](https://mlsanity.com/paper/2011.14781/) (2020)
- [Vector Transport-Free SVRG with General Retraction for Riemannian Optimization: Complexity Analysis and Practical Implementation](https://mlsanity.com/paper/1705.09059/) (2017)

## Built on by (most cited first, 2 of 2)

- [A Double Tracking Method for Optimization with Decentralized Generalized Orthogonality Constraints](https://mlsanity.com/paper/2409.04998/) (2024)
- [Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimization](https://mlsanity.com/paper/2607.01755/) (2026)

## Abstract (arXiv)

Decentralized optimization with orthogonality constraints is found widely in scientific computing and data science. Since the orthogonality constraints are nonconvex, it is quite challenging to design efficient algorithms. Existing approaches leverage the geometric tools from Riemannian optimization to solve this problem at the cost of high sample and communication complexities. To relieve this difficulty, based on two novel techniques that can waive the orthogonality constraints, we propose a variance-reduced stochastic gradient tracking (VRSGT) algorithm with the convergence rate of O(1 / k) to a stationary point. To the best of our knowledge, VRSGT is the first algorithm for decentralized optimization with orthogonality constraints that reduces both sampling and communication complexities simultaneously. In the numerical experiments, VRSGT has a promising performance in a real-world autonomous driving application.