mlsanityData to 14 Oct 2026

DR-DSGD: A Distributionally Robust Decentralized Learning Algorithm over Graphs

arXiv 2208.13810 ↗First posted 29 Aug 2022cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose to solve a regularized distributionally robust learning problem in the decentralized setting, taking into account the data distribution shift. By adding a Kullback-Liebler regularization function to the robust min-max optimization problem, the learning problem can be reduced to a modified robust minimization problem and solved efficiently. Leveraging the newly formulated optimization problem, we propose a robust version of Decentralized Stochastic Gradient Descent (DSGD), coined Distributionally Robust Decentralized Stochastic Gradient Descent (DR-DSGD). Under some mild assumptions and provided that the regularization parameter is larger than one, we theoretically prove that DR-DSGD achieves a convergence rate of Oleft(1/√KT + K/Tright), where K is the number of devices and T is the number of iterations. Simulation results show that our proposed algorithm can improve the worst distribution test accuracy by up to 10%. Moreover, DR-DSGD is more communication-efficient than DSGD since it requires fewer communication rounds (up to 20 times less) to achieve the same worst distribution test accuracy target. Furthermore, the conducted experiments reveal that DR-DSGD results in a fairer performance across devices in terms of test accuracy.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 29 Aug 2022.

Builds on

36 of 57 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  8. 2018Federated Learning with Non-IID Datacited by 1,108
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  10. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  11. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  12. 2019Agnostic Federated Learningcited by 456
24 more
  1. 2019Federated Machine Learning: Concept and Applicationscited by 412
  2. 2019Fair Resource Allocation in Federated Learningcited by 394
  3. 2018Analyzing Federated Learning through an Adversarial Lenscited by 351
  4. 2019Robust Aggregation for Federated Learningcited by 317
  5. 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 291
  6. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  7. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  8. 2013On the Convergence of Decentralized Gradient Descentcited by 159
  9. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  10. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  11. 2020Tilted Empirical Risk Minimizationcited by 106
  12. 2021Distributionally Robust Federated Averagingcited by 104
  13. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  14. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 95
  15. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  16. 2019MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Samplingcited by 67
  17. 2013On the Linear Convergence of the ADMM in Decentralized Consensus Optimizationcited by 57
  18. 2016Accelerating Stochastic Composition Optimizationcited by 46
  19. 2016On Nonconvex Decentralized Gradient Descentcited by 42
  20. 2018Robust Optimization over Multiple Domainscited by 34
  21. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 21
  22. 2021Compositional federated learning: Applications in distributionally robust averaging and meta learningcited by 12
  23. 2022Communication-Efficient Distributionally Robust Decentralized Learningcited by 10
  24. 2021Learning, Computing, and Trustworthiness in Intelligent IoT Environments: Performance-Energy Tradeoffscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2024Vanishing Variance Problem in Fully Decentralized Neural-Network Systemscited by 3
  2. 2024Communication-Efficient Federated Group Distributionally Robust Optimizationcited by 3
  3. 2023Balancing Energy Efficiency and Distributional Robustness in Over-the-Air Federated Learningcited by 1
  4. 2025Distributionally Robust Federated Learning with Client Drift Minimizationcited by 1
  5. 2026Communication-Efficient Agnostic Federated Learning via Faster Convergence and Compressioncited by 0