mlsanityData to 7 Oct 2026

Distributionally Robust Federated Averaging

arXiv 2102.12660 ↗First posted 25 Feb 2021cs.LG cs.DC stat.MLcited by 104

Not on the map. Why?

Abstract (arXiv)

In this paper, we study communication efficient distributed algorithms for distributionally robust federated learning via periodic averaging with adaptive sampling. In contrast to standard empirical risk minimization, due to the minimax structure of the underlying optimization problem, a key difficulty arises from the fact that the global parameter that controls the mixture of local losses can only be updated infrequently on the global stage. To compensate for this, we propose a Distributionally Robust Federated Averaging (DRFA) algorithm that employs a novel snapshotting scheme to approximate the accumulation of history gradients of the mixing parameter. We analyze the convergence rate of DRFA in both convex-linear and nonconvex-linear settings. We also generalize the proposed idea to objectives with regularization on the mixture parameter and propose a proximal variant, dubbed as DRFA-Prox, with provable convergence rates. We also analyze an alternative optimization method for regularized cases in strongly-convex-strongly-concave and non-convex (under PL condition)-strongly-concave settings. To the best of our knowledge, this paper is the first to solve distributionally robust federated learning with reduced communication, and to analyze the efficiency of local descent methods on distributed minimax problems. We give corroborating experimental evidence for our theoretical results in federated learning settings.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Builds on 33 of its 52 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

33 of 52 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2018LEAF: A Benchmark for Federated Settingscited by 780
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  11. 2019Agnostic Federated Learningcited by 451
  12. 2019Fair Resource Allocation in Federated Learningcited by 389
21 more
  1. 2020Adaptive Personalized Federated Learningcited by 353
  2. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  3. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 266
  4. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  5. 2017Training GANs with Optimismcited by 217
  6. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  7. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 159
  8. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  9. 2016Variance-based regularization with convex objectivescited by 153
  10. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  11. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  12. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  13. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  14. 2020Near-Optimal Algorithms for Minimax Optimizationcited by 119
  15. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  16. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  17. 2019Efficient Algorithms for Smooth Minimax Optimizationcited by 75
  18. 2018Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networkscited by 68
  19. 2017Learning with Average Top-k Losscited by 52
  20. 2019A Decentralized Parallel Algorithm for Training Generative Adversarial Netscited by 43
  21. 2019Communication-Efficient Local Decentralized SGD Methodscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 27; 2023: 23; 2024: 16; 2025: 14; 2026: 6.

104 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2022Federated Learning with Partial Model Personalizationcited by 114
  3. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  4. 2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 80
  5. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  6. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  7. 2023Fair Federated Medical Image Segmentation via Client Contribution Estimationcited by 50
  8. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  9. 2022Federated Minimax Optimization: Improved Convergence Analyses and Algorithmscited by 37
  10. 2022FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reductioncited by 35
  11. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  12. 2022Minimax Demographic Group Fairness in Federated Learningcited by 24
38 more
  1. 2021EasyFL: A Low-code Federated Learning Platform For Dummiescited by 20
  2. 2022Local Stochastic Bilevel Optimization with Momentum-Based Variance Reductioncited by 20
  3. 2022Federated Adversarial Learning: A Framework with Convergence Analysiscited by 17
  4. 2022Stochastic Constrained DRO with a Complexity Independent of Sample Sizecited by 16
  5. 2021On Bridging Generic and Personalized Federated Learning for Image Classificationcited by 15
  6. 2021Collaboration Equilibrium in Federated Learningcited by 15
  7. 2021Unified Group Fairness on Federated Learningcited by 15
  8. 2023FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clusteringcited by 15
  9. 2022Distributed Distributionally Robust Optimization with Non-Convex Objectivescited by 14
  10. 2024Unlearning during Learning: An Efficient Federated Machine Unlearning Methodcited by 14
  11. 2021AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimizationcited by 13
  12. 2023Optimal Multi-Distribution Learningcited by 13
  13. 2021Compositional federated learning: Applications in distributionally robust averaging and meta learningcited by 12
  14. 2022An Optimal Transport Approach to Personalized Federated Learningcited by 12
  15. 2023Bias Propagation in Federated Learningcited by 12
  16. 2021FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learningcited by 10
  17. 2022Communication-Efficient Distributionally Robust Decentralized Learningcited by 10
  18. 2022FeDXL: Provable Federated Learning for Deep X-Risk Optimizationcited by 10
  19. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  20. 2025Tackling Feature and Sample Heterogeneity in Decentralized Multi-Task Learning: A Sheaf-Theoretic Approachcited by 10
  21. 2021Federating for Learning Group Fair Modelscited by 9
  22. 2022A Communication-efficient Algorithm with Linear Convergence for Federated Minimax Learningcited by 9
  23. 2022On the Generalization of Wasserstein Robust Federated Learningcited by 9
  24. 2023Federated learning for secure development of AI models for Parkinson's disease detection using speech from different languagescited by 9
  25. 2023Federated Minimax Optimization with Client Heterogeneitycited by 8
  26. 2024Robust Decentralized Learning with Local Updates and Gradient Trackingcited by 8
  27. 2021A Wasserstein Minimax Framework for Mixed Linear Regressioncited by 7
  28. 2021Is Shapley Value fair? Improving Client Selection for Mavericks in Federated Learningcited by 7
  29. 2024Beyond the Federation: Topology-aware Federated Learning for Generalization to Unseen Clientscited by 7
  30. 2021Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learningcited by 6
  31. 2023Enhanced Adaptive Gradient Algorithms for Nonconvex-PL Minimax Optimizationcited by 6
  32. 2023Scaff-PD: Communication Efficient Fair and Robust Federated Learningcited by 6
  33. 2023Distributed Personalized Empirical Risk Minimizationcited by 6
  34. 2024Trust the Process: Zero-Knowledge Machine Learning to Enhance Trust in Generative AI Interactionscited by 6
  35. 2022Learning Distributionally Robust Models at Scale via Composite Optimizationcited by 5
  36. 2022DR-DSGD: A Distributionally Robust Decentralized Learning Algorithm over Graphscited by 5
  37. 2022Adaptive Federated Minimax Optimization with Lower Complexitiescited by 5
  38. 2023Mitigating Group Bias in Federated Learning for Heterogeneous Devicescited by 5

50 most cited of 104 All 104 (JSON)