mlsanityData to 7 Oct 2026

Decentralized Gradient Tracking with Local Steps

arXiv 2301.01313 ↗First posted 3 Jan 2023math.OC cs.DC cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Gradient tracking (GT) is an algorithm designed for solving decentralized optimization problems over a network (such as training a machine learning model). A key feature of GT is a tracking mechanism that allows to overcome data heterogeneity between nodes. We develop a novel decentralized tracking mechanism, K-GT, that enables communication-efficient local updates in GT while inheriting the data-independence property of GT. We prove a convergence rate for K-GT on smooth non-convex functions and prove that it reduces the communication overhead asymptotically by a linear factor K, where K denotes the number of local steps. We illustrate the robustness and effectiveness of this heterogeneity correction on convex and non-convex benchmark problems and on a non-convex neural network training task with the MNIST dataset.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 3 Jan 2023.

Builds on

26 of 40 references.

References · 26

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  8. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  9. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  11. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  12. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
14 more
  1. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  2. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  3. 2016Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphscited by 149
  4. 2018D2: Decentralized Training over Decentralized Datacited by 140
  5. 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 109
  6. 2016NEXT: In-Network Nonconvex Optimizationcited by 107
  7. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  8. 2022An Improved Analysis of Gradient Tracking for Decentralized Machine Learningcited by 68
  9. 2019Variance-Reduced Decentralized Stochastic Optimization with Accelerated Convergencecited by 33
  10. 2021Accelerating Gossip SGD with Periodic Global Averagingcited by 25
  11. 2022On the Performance of Gradient Tracking with Local Updatescited by 21
  12. 2019Linear Convergence of Primal-Dual Gradient Methods and their Performance in Distributed Optimizationcited by 11
  13. 2020Can Primal Methods Outperform Primal-dual Methods in Decentralized Dynamic Optimization?cited by 10
  14. 2018A Distributed Stochastic Gradient Tracking Methodcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 11; 2026: 6.

32 papers.

Sort:

  1. 2023DIGEST: Fast and Communication Efficient Decentralized Learning with Local Updatescited by 12
  2. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 12
  3. 2024Robust Decentralized Learning with Local Updates and Gradient Trackingcited by 8
  4. 2024The Effectiveness of Local Updates for Decentralized Learning under Data Heterogeneitycited by 7
  5. 2024Hierarchical Federated Learning with Multi-Timescale Gradient Correctioncited by 7
  6. 2023Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updatescited by 4
  7. 2023On the Computation-Communication Trade-Off with A Flexible Gradient Tracking Approachcited by 4
  8. 2024Differentiated Aggregation to Improve Generalization in Federated Learningcited by 4
  9. 2024Differential error feedback for communication-efficient decentralized learningcited by 3
  10. 2025Communication-Efficient Stochastic Distributed Learningcited by 3
  11. 2023Communication-Efficient Federated Optimization over Semi-Decentralized Networkscited by 2
  12. 2023A Proximal Gradient Method With Probabilistic Multi-Gossip Communications for Decentralized Composite Optimizationcited by 2
20 more
  1. 2023Achieving Linear Speedup with ProxSkip in Distributed Stochastic Optimizationcited by 1
  2. 2025Gradient Correction in Federated Learning with Adaptive Optimizationcited by 1
  3. 2025Exploiting Similarity for Computation and Communication-Efficient Decentralized Optimizationcited by 1
  4. 2025Jointly Computation- and Communication-Efficient Distributed Learningcited by 1
  5. 2025A Communication-Efficient Decentralized Actor-Critic Algorithmcited by 1
  6. 2026Subspace Optimization for Efficient Federated Learning under Heterogeneous Datacited by 1
  7. 2024MUSIC: Accelerated Convergence for Distributed Optimization With Inexact and Exact Methodscited by 0
  8. 2024Fast Decentralized Gradient Tracking for Federated Minimax Optimization with Local Updatescited by 0
  9. 2024Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Stepscited by 0
  10. 2024A Stochastic Approximation Approach for Efficient Decentralized Optimization on Random Networkscited by 0
  11. 2025Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentumcited by 0
  12. 2025Diffusion Learning with Partial Agent Participation and Local Updatescited by 0
  13. 2025CoCoL: A Communication Efficient Decentralized Collaborative Method for Multi-Robot Systemscited by 0
  14. 2025Pareto-optimal Trade-offs Between Communication and Computation with Flexible Gradient Trackingcited by 0
  15. 2025Beyond Scaffold: A Unified Spatio-Temporal Gradient Tracking Methodcited by 0
  16. 2026Accelerating Decentralized Optimization via Overlapping Local Stepscited by 0
  17. 2026Local Updates in Distributed Optimization: Provable Acceleration and Topology Effectscited by 0
  18. 2026Sporadic Gradient Tracking over Directed Graphs: A Theoretical Perspective on Decentralized Federated Learningcited by 0
  19. 2026Local adapt-then-combine algorithms for distributed nonsmooth optimization: Achieving provable communication accelerationcited by 0
  20. 2026Decentralized Learning via Random Walk with Jumpscited by 0