mlsanityData to 14 Oct 2026

When Do Curricula Work in Federated Learning?

arXiv 2212.12712 ↗First posted 24 Dec 2022cs.LG cs.AI cs.DCcited by 9

Not on the map. Why?

Abstract (arXiv)

An oft-cited open problem of federated learning is the existence of data heterogeneity at the clients. One pathway to understanding the drastic accuracy drop in federated learning is by scrutinizing the behavior of the clients' deep models on data with different levels of "difficulty", which has been left unaddressed. In this paper, we investigate a different and rarely studied dimension of FL: ordered learning. Specifically, we aim to investigate how ordered learning principles can contribute to alleviating the heterogeneity effects in FL. We present theoretical analysis and conduct extensive empirical studies on the efficacy of orderings spanning three kinds of learning: curriculum, anti-curriculum, and random curriculum. We find that curriculum learning largely alleviates non-IIDness. Interestingly, the more disparate the data distributions across clients the more they benefit from ordered learning. We provide analysis explaining this phenomenon, specifically indicating how curriculum training appears to make the objective landscape progressively less convex, suggesting fast converging iterations at the beginning of the training procedure. We derive quantitative results of convergence for both convex and nonconvex objectives by modeling the curriculum training on federated devices as local SGD with locally biased stochastic gradients. Also, inspired by ordered learning, we propose a novel client selection technique that benefits from the real-world disparity in the clients. Our proposed approach to client selection has a synergic effect when applied together with ordered learning in FL.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 24 Dec 2022.

Builds on

35 of 47 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  7. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  8. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  9. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  10. 2020Adaptive Personalized Federated Learningcited by 358
  11. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  12. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
23 more
  1. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  2. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 198
  3. 2021ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detectioncited by 159
  4. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  5. 2018CurriculumNet: Weakly Supervised Learning from Large-Scale Web Imagescited by 126
  6. 2020Optimal Client Sampling for Federated Learningcited by 124
  7. 2020When Do Curricula Work?cited by 106
  8. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 92
  9. 2020On the Convergence of SGD with Biased Gradientscited by 83
  10. 2014Curriculum Learning of Multiple Taskscited by 66
  11. 2019Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 59
  12. 2019Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narrativescited by 59
  13. 2018An Empirical Exploration of Curriculum Learning for Neural Machine Translationcited by 58
  14. 2019Non-asymptotic Analysis of Biased Stochastic Approximation Schemecited by 45
  15. 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
  16. 2020Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Searchcited by 33
  17. 2022A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learningcited by 31
  18. 2021Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneitycited by 27
  19. 2016Self Paced Deep Learning for Weakly Supervised Object Detectioncited by 25
  20. 2022Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarkscited by 24
  21. 2022FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distributioncited by 22
  22. 2021Memory-aware curriculum federated learning for breast cancer classificationcited by 20
  23. 2020Budgeted Online Selection of Candidate IoT Clients to Participate in Federated Learningcited by 14

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3; 2025: 2.

9 papers.

  1. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  2. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  3. 2023Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latentscited by 16
  4. 2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
  5. 2024Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updatescited by 4
  6. 2025Curriculum Guided Personalized Subgraph Federated Learningcited by 4
  7. 2023CEFHRI: A Communication Efficient Federated Learning Framework for Recognizing Industrial Human-Robot Interactioncited by 3
  8. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  9. 2024Personalized Federated Learning via Sequential Layer Expansion in Representation Learningcited by 0