Communication-Efficient Federated Group Distributionally Robust Optimization
Not on the map. Why?
Abstract (arXiv)
Federated learning faces challenges due to the heterogeneity in data volumes and distributions at different clients, which can compromise model generalization ability to various distributions. Existing approaches to address this issue based on group distributionally robust optimization (GDRO) often lead to high communication and sample complexity. To this end, this work introduces algorithms tailored for communication-efficient Federated Group Distributionally Robust Optimization (FGDRO). Our contributions are threefold: Firstly, we introduce the FGDRO-CVaR algorithm, which optimizes the average top-K losses while reducing communication complexity to O(1/ε4), where ε denotes the desired precision level. Secondly, our FGDRO-KL algorithm is crafted to optimize KL regularized FGDRO, cutting communication complexity to O(1/ε3). Lastly, we propose FGDRO-KL-Adam to utilize Adam-type local updates in FGDRO-KL, which not only maintains a communication cost of O(1/ε3) but also shows potential to surpass SGD-type local steps in practical applications. The effectiveness of our algorithms has been demonstrated on a variety of real-world tasks, including natural language processing and computer vision.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 65 of its 83 references that are in the map. First posted on arXiv 8 Oct 2024.
Builds on
65 of 83 references.
References · 65
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2021Convex optimizationcited by 2,479
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
- 2020Adaptive Federated Optimizationcited by 1,036
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
- 2018Local SGD Converges Fast and Communicates Littlecited by 595
38 more
- 2018Federated Learning for Mobile Keyboard Predictioncited by 565
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
- 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
- 2019Agnostic Federated Learningcited by 451
- 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
- 2018Sparsified SGD with Memorycited by 373
- 2020Adaptive Personalized Federated Learningcited by 353
- 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
- 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
- 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
- 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
- 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
- 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
- 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
- 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
- 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
- 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
- 2018Stochastic model-based minimization of weakly convex functionscited by 157
- 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
- 2016Variance-based regularization with convex objectivescited by 153
- 2020Is Local SGD Better than Minibatch SGD?cited by 151
- 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
- 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
- 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112
- 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
- 2021Distributionally Robust Federated Averagingcited by 104
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
- 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 93
- 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
- 2022Federated Learning Enables Big Data for Rare Cancer Boundary Detectioncited by 83
- 2021The iWildCam 2021 Competition Datasetcited by 77
- 2016CoCoA: A General Framework for Communication-Efficient Distributed Optimizationcited by 69
- 2021What Do We Mean by Generalization in Federated Learning?cited by 62
- 2017Learning with Average Top-k Losscited by 52
- 2021A Novel Convergence Analysis for Algorithms of the Adam Familycited by 50
- 2020An Online Method for A Class of Distributionally Robust Optimization with Non-Convex Objectivescited by 42
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.