# When Do Curricula Work in Federated Learning?

arXiv: https://arxiv.org/abs/2212.12712 · HTML: https://mlsanity.com/paper/2212.12712/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 24 Dec 2022.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Tighter Theory for Local SGD on Identical and Heterogeneous Data](https://mlsanity.com/paper/1909.04746/) (2019)
- [Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies](https://mlsanity.com/paper/2010.01243/) (2020)
- [ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2103.05346/) (2021)
- [On the Convergence of Local Descent Methods in Federated Learning](https://mlsanity.com/paper/1910.14425/) (2019)
- [CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images](https://mlsanity.com/paper/1808.01097/) (2018)
- [Optimal Client Sampling for Federated Learning](https://mlsanity.com/paper/2010.13723/) (2020)
- [When Do Curricula Work?](https://mlsanity.com/paper/2012.03107/) (2020)
- [On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization](https://mlsanity.com/paper/1708.01012/) (2017)
- [On the Convergence of SGD with Biased Gradients](https://mlsanity.com/paper/2008.00051/) (2020)
- [Curriculum Learning of Multiple Tasks](https://mlsanity.com/paper/1412.1353/) (2014)
- [Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/1905.05816/) (2019)
- [Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives](https://mlsanity.com/paper/1905.10847/) (2019)
- [An Empirical Exploration of Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1811.00739/) (2018)
- [Non-asymptotic Analysis of Biased Stochastic Approximation Scheme](https://mlsanity.com/paper/1902.00629/) (2019)
- [Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces](https://mlsanity.com/paper/2209.10526/) (2022)
- [Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search](https://mlsanity.com/paper/2007.07197/) (2020)
- [A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learning](https://mlsanity.com/paper/2201.02932/) (2022)
- [Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity](https://mlsanity.com/paper/2105.00562/) (2021)

## Built on by (most cited first, 9 of 9)

- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training](https://mlsanity.com/paper/2304.06947/) (2023)
- [Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents](https://mlsanity.com/paper/2312.01537/) (2023)
- [Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models](https://mlsanity.com/paper/2410.00131/) (2024)
- [Stragglers-Aware Low-Latency Synchronous Federated Learning via Layer-Wise Model Updates](https://mlsanity.com/paper/2403.18375/) (2024)
- [Curriculum Guided Personalized Subgraph Federated Learning](https://mlsanity.com/paper/2509.00402/) (2025)
- [CEFHRI: A Communication Efficient Federated Learning Framework for Recognizing Industrial Human-Robot Interaction](https://mlsanity.com/paper/2308.14965/) (2023)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [Personalized Federated Learning via Sequential Layer Expansion in Representation Learning](https://mlsanity.com/paper/2404.17799/) (2024)

## Abstract (arXiv)

An oft-cited open problem of federated learning is the existence of data heterogeneity at the clients. One pathway to understanding the drastic accuracy drop in federated learning is by scrutinizing the behavior of the clients' deep models on data with different levels of "difficulty", which has been left unaddressed. In this paper, we investigate a different and rarely studied dimension of FL: ordered learning. Specifically, we aim to investigate how ordered learning principles can contribute to alleviating the heterogeneity effects in FL. We present theoretical analysis and conduct extensive empirical studies on the efficacy of orderings spanning three kinds of learning: curriculum, anti-curriculum, and random curriculum. We find that curriculum learning largely alleviates non-IIDness. Interestingly, the more disparate the data distributions across clients the more they benefit from ordered learning. We provide analysis explaining this phenomenon, specifically indicating how curriculum training appears to make the objective landscape progressively less convex, suggesting fast converging iterations at the beginning of the training procedure. We derive quantitative results of convergence for both convex and nonconvex objectives by modeling the curriculum training on federated devices as local SGD with locally biased stochastic gradients. Also, inspired by ordered learning, we propose a novel client selection technique that benefits from the real-world disparity in the clients. Our proposed approach to client selection has a synergic effect when applied together with ordered learning in FL.