# Non-IID data and Continual Learning processes in Federated Learning: A long road ahead

arXiv: https://arxiv.org/abs/2111.13394 · HTML: https://mlsanity.com/paper/2111.13394/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 83 of its 159 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 30 of 83)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)

## Built on by (most cited first, 30 of 36)

- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [Federated Continual Learning via Knowledge Fusion: A Survey](https://mlsanity.com/paper/2312.16475/) (2023)
- [Topology-aware Generalization of Decentralized SGD](https://mlsanity.com/paper/2206.12680/) (2022)
- [Towards Efficient Replay in Federated Incremental Learning](https://mlsanity.com/paper/2403.05890/) (2024)
- [Integration of Large Language Models and Federated Learning](https://mlsanity.com/paper/2307.08925/) (2023)
- [Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning](https://mlsanity.com/paper/2409.01128/) (2024)
- [Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey](https://mlsanity.com/paper/2412.13840/) (2024)
- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [Federated Continual Learning: Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2502.07059/) (2025)
- [Spatial-Temporal Federated Learning for Lifelong Person Re-identification on Distributed Edges](https://mlsanity.com/paper/2207.11759/) (2022)
- [Non-readily identifiable data collaboration analysis for multiple datasets including personal information](https://mlsanity.com/paper/2208.14611/) (2022)
- [FedProK: Trustworthy Federated Class-Incremental Learning via Prototypical Feature Knowledge Transfer](https://mlsanity.com/paper/2405.02685/) (2024)
- [Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling](https://mlsanity.com/paper/2410.15143/) (2024)
- [Federated Continual Learning for Socially Aware Robotics](https://mlsanity.com/paper/2201.05527/) (2022)
- [A Thorough Assessment of the Non-IID Data Impact in Federated Learning](https://mlsanity.com/paper/2503.17070/) (2025)
- [Review of Mathematical Optimization in Federated Learning](https://mlsanity.com/paper/2412.01630/) (2024)
- [AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Data](https://mlsanity.com/paper/2505.12245/) (2025)
- [Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontiers](https://mlsanity.com/paper/2511.22616/) (2025)
- [Decentralized digital twins of complex dynamical systems](https://mlsanity.com/paper/2207.12245/) (2022)
- [An Information-Theoretic Analysis for Federated Learning under Concept Drift](https://mlsanity.com/paper/2506.21036/) (2025)
- [Masked Autoencoders are Efficient Continual Federated Learners](https://mlsanity.com/paper/2306.03542/) (2023)
- [Fingerprint Attack: Client De-Anonymization in Federated Learning](https://mlsanity.com/paper/2310.05960/) (2023)
- [Estimation of conditional average treatment effects on distributed confidential data](https://mlsanity.com/paper/2402.02672/) (2024)
- [Variational Bayes for Federated Continual Learning](https://mlsanity.com/paper/2405.14291/) (2024)
- [FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classification](https://mlsanity.com/paper/2503.18981/) (2025)
- [FedTeddi: Temporal Drift and Divergence Aware Scheduling for Timely Federated Edge Learning](https://mlsanity.com/paper/2509.07342/) (2025)
- [Federated continual learning: A comprehensive survey on lifelong and privacy-preserving learning over distributed and non-stationary data](https://mlsanity.com/paper/2606.11272/) (2026)
- [C2FL: Clustered Continual Federated Learning under Spatial and Temporal Drift](https://mlsanity.com/paper/2606.18003/) (2026)
- [Global Layers: Non-IID Tabular Federated Learning](https://mlsanity.com/paper/2305.19290/) (2023)
- [Towards Privacy-Aware and Personalised Assistive Robots: A User-Centred Approach](https://mlsanity.com/paper/2405.14528/) (2024)

## Abstract (arXiv)

Federated Learning is a novel framework that allows multiple devices or institutions to train a machine learning model collaboratively while preserving their data private. This decentralized approach is prone to suffer the consequences of data statistical heterogeneity, both across the different entities and over time, which may lead to a lack of convergence. To avoid such issues, different methods have been proposed in the past few years. However, data may be heterogeneous in lots of different ways, and current proposals do not always determine the kind of heterogeneity they are considering. In this work, we formally classify data statistical heterogeneity and review the most remarkable learning strategies that are able to face it. At the same time, we introduce approaches from other machine learning frameworks, such as Continual Learning, that also deal with data heterogeneity and could be easily adapted to the Federated Learning settings.