# A general framework for decentralized optimization with first-order methods

arXiv: https://arxiv.org/abs/2009.05837 · HTML: https://mlsanity.com/paper/2009.05837/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2020.

## Built on by (most cited first, 28 of 28)

- [A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimization](https://mlsanity.com/paper/2102.06752/) (2021)
- [DESTRESS: Computation-Optimal and Communication-Efficient Decentralized Nonconvex Finite-Sum Optimization](https://mlsanity.com/paper/2110.01165/) (2021)
- [Unified Analysis of Decentralized Gradient Descent: a Contraction Mapping Framework](https://mlsanity.com/paper/2503.14353/) (2025)
- [Distributed support-vector-machine over dynamic balanced directed networks](https://mlsanity.com/paper/2104.00399/) (2021)
- [Non-Coherent Over-the-Air Decentralized Gradient Descent](https://mlsanity.com/paper/2211.10777/) (2022)
- [Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compression](https://mlsanity.com/paper/2305.09896/) (2023)
- [Efficiency Ordering of Stochastic Gradient Descent](https://mlsanity.com/paper/2209.07446/) (2022)
- [A survey on secure decentralized optimization and learning](https://mlsanity.com/paper/2408.08628/) (2024)
- [Distributed saddle point problems for strongly concave-convex functions](https://mlsanity.com/paper/2202.05812/) (2022)
- [Decentralized Learning over Wireless Networks: The Effect of Broadcast with Random Access](https://mlsanity.com/paper/2305.07368/) (2023)
- [Variance reduced stochastic optimization over directed graphs with row and column stochastic weights](https://mlsanity.com/paper/2202.03346/) (2022)
- [A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints](https://mlsanity.com/paper/2208.13643/) (2022)
- [Federated Learning in Wireless Networks via Over-the-Air Computations](https://mlsanity.com/paper/2305.04630/) (2023)
- [Robust Online Learning over Networks](https://mlsanity.com/paper/2309.00520/) (2023)
- [Limited Communications Distributed Optimization via Deep Unfolded Distributed ADMM](https://mlsanity.com/paper/2309.14353/) (2023)
- [Faster Convergence with Less Communication: Broadcast-Based Subgraph Sampling for Decentralized Learning over Wireless Networks](https://mlsanity.com/paper/2401.13779/) (2024)
- [A Survey on Distributed Online Optimization and Game](https://mlsanity.com/paper/2205.00473/) (2022)
- [A Bias-Correction Decentralized Stochastic Gradient Algorithm with Momentum Acceleration](https://mlsanity.com/paper/2501.19082/) (2025)
- [Distributed Finite-Sum Constrained Optimization subject to Nonlinearity on the Node Dynamics](https://mlsanity.com/paper/2203.14527/) (2022)
- [A Unified Framework for Center-based Clustering of Distributed Data](https://mlsanity.com/paper/2402.01302/) (2024)
- [The Artificial Neural Twin -- Process Optimization and Continual Learning in Distributed Process Chains](https://mlsanity.com/paper/2403.18343/) (2024)
- [Distributed Learning over Arbitrary Topology: Linear Speed-Up with Polynomial Transient Time](https://mlsanity.com/paper/2503.16123/) (2025)
- [Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum](https://mlsanity.com/paper/2504.12742/) (2025)
- [Decentralized Stochastic Nonconvex Optimization under the (L0,L1)-Smoothness](https://mlsanity.com/paper/2509.08726/) (2025)
- [Distributed Dynamic Associative Memory via Online Convex Optimization](https://mlsanity.com/paper/2511.23347/) (2025)
- [Sporadic Gradient Tracking over Directed Graphs: A Theoretical Perspective on Decentralized Federated Learning](https://mlsanity.com/paper/2602.00791/) (2026)
- [Adaptive Decentralized Composite Optimization via Three-Operator Splitting](https://mlsanity.com/paper/2602.17545/) (2026)
- [High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking](https://mlsanity.com/paper/2605.00281/) (2026)

## Abstract (arXiv)

Decentralized optimization to minimize a finite sum of functions over a network of nodes has been a significant focus within control and signal processing research due to its natural relevance to optimal control and signal estimation problems. More recently, the emergence of sophisticated computing and large-scale data science needs have led to a resurgence of activity in this area. In this article, we discuss decentralized first-order gradient methods, which have found tremendous success in control, signal processing, and machine learning problems, where such methods, due to their simplicity, serve as the first method of choice for many complex inference and training tasks. In particular, we provide a general framework of decentralized first-order methods that is applicable to undirected and directed communication networks alike, and show that much of the existing work on optimization and consensus can be related explicitly to this framework. We further extend the discussion to decentralized stochastic first-order methods that rely on stochastic gradients at each node and describe how local variance reduction schemes, previously shown to have promise in the centralized settings, are able to improve the performance of decentralized methods when combined with what is known as gradient tracking. We motivate and demonstrate the effectiveness of the corresponding methods in the context of machine learning and signal processing problems that arise in decentralized environments.