mlsanityData to 7 Oct 2026

Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

arXiv 2102.04761 ↗First posted 9 Feb 2021cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge and may severely deteriorate the generalization performance. In this paper, we investigate and identify the limitation of several decentralized optimization algorithms for different degrees of data heterogeneity. We propose a novel momentum-based method to mitigate this decentralized training difficulty. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10, ImageNet, and AG News) and several network topologies (Ring and Social Network) that our method is much more robust to the heterogeneity of clients' data than other existing methods, by a significant improvement in test performance (1% !-! 20%). Our code is publicly available.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 71 of its 90 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

71 of 90 references.

References · 71

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2019Advances and Open Problems in Federated Learningcited by 3,021
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2018Group Normalizationcited by 1,514
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
38 more
  1. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2018Federated Learning with Non-IID Datacited by 1,099
  4. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  7. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  8. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  9. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  10. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  11. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  12. 2018How Does Batch Normalization Help Optimization?cited by 355
  13. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  14. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  15. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  16. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  17. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  18. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  19. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  20. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  21. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  22. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  23. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  24. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 159
  25. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  26. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  27. 2018D2: Decentralized Training over Decentralized Datacited by 140
  28. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  29. 2017Network Topology and Communication-Computation Tradeoffs in Decentralized Optimizationcited by 119
  30. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  31. 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 109
  32. 2018Distributed Stochastic Gradient Tracking Methodscited by 106
  33. 2018Communication Compression for Decentralized Trainingcited by 87
  34. 2017Optimal algorithms for smooth and strongly convex distributed optimization in networkscited by 81
  35. 2019MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Samplingcited by 67
  36. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  37. 2020Siloed Federated Learning for Multi-Centric Histopathology Datasetscited by 61
  38. 2018Towards Understanding Regularization in Batch Normalizationcited by 56

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 20; 2023: 16; 2024: 10; 2025: 6; 2026: 5.

69 papers.

Sort:

  1. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  2. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  3. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  4. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  5. 2022An Improved Analysis of Gradient Tracking for Decentralized Machine Learningcited by 68
  6. 2021Exponential Graph is Provably Efficient for Decentralized Deep Trainingcited by 55
  7. 2021DecentLaM: Decentralized Momentum SGD for Large-batch Deep Trainingcited by 40
  8. 2021RelaySum for Decentralized Deep Learning on Heterogeneous Datacited by 39
  9. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  10. 2022Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Datacited by 31
  11. 2022Byzantine-Robust Decentralized Learning via ClippedGossipcited by 29
  12. 2021Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGDcited by 28
38 more
  1. 2022Beyond spectral gap: The role of the topology in decentralized learningcited by 27
  2. 2021Personalized Federated Learning with Adaptive Batchnorm for Healthcarecited by 26
  3. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  4. 2021Decentralized Local Stochastic Extra-Gradient for Variational Inequalitiescited by 25
  5. 2022Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matchingcited by 24
  6. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  7. 2022Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimizationcited by 22
  8. 2022Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Datacited by 20
  9. 2021BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learningcited by 16
  10. 2023Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?cited by 14
  11. 2021D-Cliques: Compensating for Data Heterogeneity with Topology in Decentralized Federated Learningcited by 12
  12. 2022Data-heterogeneity-aware Mixing for Decentralized Learningcited by 12
  13. 2023Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Datacited by 12
  14. 2024Decentralized Directed Collaboration for Personalized Federated Learningcited by 9
  15. 2022On Generalization of Decentralized Learning with Separable Datacited by 8
  16. 2023Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergencecited by 8
  17. 2023Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Trainingcited by 8
  18. 2023A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learningcited by 7
  19. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 7
  20. 2023DSGD-CECA: Decentralized SGD with Communication-Optimal Exact Consensus Algorithmcited by 6
  21. 2022Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compressioncited by 5
  22. 2022Neighborhood Gradient Clustering: An Efficient Decentralized Learning Method for Non-IID Data Distributionscited by 5
  23. 2023Asymmetrically Decentralized Federated Learningcited by 5
  24. 2023DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representationscited by 5
  25. 2024Ordered Momentum for Asynchronous SGDcited by 5
  26. 2022Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networkscited by 4
  27. 2024From promise to practice: realizing high-performance decentralized trainingcited by 4
  28. 2023Beyond spectral gap (extended): The role of the topology in decentralized learningcited by 3
  29. 2023Homogenizing Non-IID datasets via In-Distribution Knowledge Distillation for Decentralized Learningcited by 3
  30. 2024DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Modelscited by 3
  31. 2025Scalable Decentralized Learning with Teleportationcited by 3
  32. 2021Low Precision Decentralized Distributed Training over IID and non-IID Datacited by 2
  33. 2023CoDeC: Communication-Efficient Decentralized Continual Learningcited by 2
  34. 2024Smart Sampling: Helping from Friendly Neighbors for Decentralized Federated Learningcited by 2
  35. 2025PDSL: Privacy-Preserved Decentralized Stochastic Learning with Heterogeneous Data Distributioncited by 2
  36. 2023Structured Cooperative Learning with Graphical Model Priorscited by 1
  37. 2024Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Datacited by 1
  38. 2024FedNMUT -- Federated Noisy Model Update Tracking Convergence Analysiscited by 1

50 most cited of 69 All 69 (JSON)