Understanding A Class of Decentralized and Federated Optimization Algorithms: A Multi-Rate Feedback Control Perspective
Not on the map. Why?
Abstract (arXiv)
Distributed algorithms have been playing an increasingly important role in many applications such as machine learning, signal processing, and control. Significant research efforts have been devoted to developing and analyzing new algorithms for various applications. In this work, we provide a fresh perspective to understand, analyze, and design distributed optimization algorithms. Through the lens of multi-rate feedback control, we show that a wide class of distributed algorithms, including popular decentralized/federated schemes, can be viewed as discretizing a certain continuous-time feedback control system, possibly with multiple sampling rates, such as decentralized gradient descent, gradient tracking, and federated averaging. This key observation not only allows us to develop a generic framework to analyze the convergence of the entire algorithm class. More importantly, it also leads to an interesting way of designing new distributed algorithms. We develop the theory behind our framework and provide examples to highlight how the framework can be used in practice.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 27 Apr 2022.
Builds on
22 of 36 references.
References · 22
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2020Adaptive Federated Optimizationcited by 1,036
- 2019Towards Federated Learning at Scale: System Designcited by 826
- 2013On the Convergence of Decentralized Gradient Descentcited by 158
- 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
- 2016NEXT: In-Network Nonconvex Optimizationcited by 107
- 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
- 2017Optimal algorithms for smooth and strongly convex distributed optimization in networkscited by 81
- 2019First Analysis of Local GD on Heterogeneous Datacited by 80
- 2017A decentralized proximal-gradient method with network independent step-sizes and separated convergence ratescited by 51
10 more
- 2019A Dynamical Systems Perspective on Nesterov Accelerationcited by 33
- 2015A geometric alternative to Nesterov's accelerated gradient descentcited by 21
- 2020Multi-consensus Decentralized Accelerated Gradient Descentcited by 20
- 2018Continuous-time Models for Stochastic Optimization Algorithmscited by 16
- 2020Can Primal Methods Outperform Primal-dual Methods in Decentralized Dynamic Optimization?cited by 10
- 2017Control Interpretations for First-Order Optimization Methodscited by 8
- 2018Distributed Non-Convex First-Order Optimization and Information Processing: Lower Complexity Bounds and Rate Optimal Algorithmscited by 6
- 2020Distributed Gradient Flow: Nonsmoothness, Nonconvexity, and Saddle Point Evasioncited by 4
- 2013Continuous-time Proportional-Integral Distributed Optimization for Networked Systemscited by 2
- 2019Distributed Gradient Descent: Nonconvergence to Saddle Points and the Stable-Manifold Theoremcited by 1
Built on by
Citing papers per year: 2024: 2.
2 papers.