# Optimized Gradient Tracking for Decentralized Online Learning

arXiv: https://arxiv.org/abs/2306.06375 · HTML: https://mlsanity.com/paper/2306.06375/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 59 references that are in the map.
- First posted on arXiv 10 Jun 2023.

## Builds on (most cited first, 26 of 26)

- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphs](https://mlsanity.com/paper/1607.03218/) (2016)
- [D2: Decentralized Training over Decentralized Data](https://mlsanity.com/paper/1803.07068/) (2018)
- [Adaptive Online Learning in Dynamic Environments](https://mlsanity.com/paper/1810.10815/) (2018)
- [A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates](https://mlsanity.com/paper/1704.07807/) (2017)
- [Improved Dynamic Regret for Non-degenerate Functions](https://mlsanity.com/paper/1608.03933/) (2016)
- [A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learning](https://mlsanity.com/paper/2110.09993/) (2021)
- [Distributed Online Optimization in Dynamic Environments Using Mirror Descent](https://mlsanity.com/paper/1609.02845/) (2016)
- [DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization](https://mlsanity.com/paper/1901.09109/) (2019)
- [Distributed Online Convex Optimization with Time-Varying Coupled Inequality Constraints](https://mlsanity.com/paper/1903.04277/) (2019)
- [Distributed Nonconvex Constrained Optimization over Time-Varying Digraphs](https://mlsanity.com/paper/1809.01106/) (2018)
- [Online Learning with Inexact Proximal Online Gradient Descent Algorithms](https://mlsanity.com/paper/1806.00202/) (2018)
- [Decentralized Proximal Gradient Algorithms with Linear Convergence Rates](https://mlsanity.com/paper/1909.06479/) (2019)
- [A Distributed Online Convex Optimization Algorithm with Improved Dynamic Regret](https://mlsanity.com/paper/1911.05050/) (2019)
- [GTAdam: Gradient Tracking with Adaptive Momentum for Distributed Online Optimization](https://mlsanity.com/paper/2009.01745/) (2020)
- [Improving Dynamic Regret in Distributed Online Mirror Descent Using Primal and Dual Information](https://mlsanity.com/paper/2112.03504/) (2021)
- [Tracking Moving Agents via Inexact Online Gradient Descent Algorithm](https://mlsanity.com/paper/1710.05133/) (2017)
- [Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods](https://mlsanity.com/paper/1810.08727/) (2018)
- [Online Learning over Dynamic Graphs via Distributed Proximal Gradient Algorithm](https://mlsanity.com/paper/1905.07018/) (2019)
- [Decentralized Online Learning: Take Benefits from Others' Data without Sharing Your Own to Track Global Trend](https://mlsanity.com/paper/1901.10593/) (2019)
- [Distributed Online Convex Optimization with Improved Dynamic Regret](https://mlsanity.com/paper/1911.05127/) (2019)
- [Explicit Convergence Rate of a Distributed Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1312.1085/) (2013)
- [A Coordinate Descent Primal-Dual Algorithm and Application to Distributed Asynchronous Optimization](https://mlsanity.com/paper/1407.0898/) (2014)
- [Dynamic and Distributed Online Convex Optimization for Demand Response of Commercial Buildings](https://mlsanity.com/paper/2002.00099/) (2020)
- [Distributed Nonconvex Optimization for Sparse Representation](https://mlsanity.com/paper/1611.06576/) (2016)

## Built on by (most cited first, 1 of 1)

- [A Hybrid Stochastic Gradient Tracking Method for Distributed Online Optimization Over Time-Varying Directed Networks](https://mlsanity.com/paper/2508.20645/) (2025)

## Abstract (arXiv)

This work considers the problem of decentralized online learning, where the goal is to track the optimum of the sum of time-varying functions, distributed across several nodes in a network. The local availability of the functions and their gradients necessitates coordination and consensus among the nodes. We put forth the Generalized Gradient Tracking (GGT) framework that unifies a number of existing approaches, including the state-of-the-art ones. The performance of the proposed GGT algorithm is theoretically analyzed using a novel semidefinite programming-based analysis that yields the desired regret bounds under very general conditions and without requiring the gradient boundedness assumption. The results are applicable to the special cases of GGT, which include various state-of-the-art algorithms as well as new dynamic versions of various classical decentralized algorithms. To further minimize the regret, we consider a condensed version of GGT with only four free parameters. A procedure for offline tuning of these parameters using only the problem parameters is also detailed. The resulting optimized GGT (oGGT) algorithm not only achieves improved dynamic regret bounds, but also outperforms all state-of-the-art algorithms on both synthetic and real-world datasets.