mlsanityData to 14 Oct 2026

Non-IID data and Continual Learning processes in Federated Learning: A long road ahead

arXiv 2111.13394 ↗First posted 26 Nov 2021cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Federated Learning is a novel framework that allows multiple devices or institutions to train a machine learning model collaboratively while preserving their data private. This decentralized approach is prone to suffer the consequences of data statistical heterogeneity, both across the different entities and over time, which may lead to a lack of convergence. To avoid such issues, different methods have been proposed in the past few years. However, data may be heterogeneous in lots of different ways, and current proposals do not always determine the kind of heterogeneity they are considering. In this work, we formally classify data statistical heterogeneity and review the most remarkable learning strategies that are able to face it. At the same time, we introduce approaches from other machine learning frameworks, such as Continual Learning, that also deal with data heterogeneity and could be easily adapted to the Federated Learning settings.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 83 of its 159 references that are in the map. First posted on arXiv 26 Nov 2021.

Builds on

83 of 159 references.

References · 83

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  9. 2016Learning without Forgettingcited by 2,496
  10. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  11. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  12. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
38 more
  1. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  3. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,329
  4. 2017Adversarial Discriminative Domain Adaptationcited by 1,311
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  6. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  7. 2017Continual Learning with Deep Generative Replaycited by 1,158
  8. 2018Federated Learning with Non-IID Datacited by 1,108
  9. 2018Experience Replay for Continual Learningcited by 1,095
  10. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  11. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  12. 2017A Survey on Multi-Task Learningcited by 840
  13. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  14. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  15. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  16. 2017Federated Multi-Task Learningcited by 672
  17. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  18. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 612
  19. 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 600
  20. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  21. 2019Federated Learning with Personalization Layerscited by 541
  22. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  23. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  24. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  25. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
  26. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  27. 2019Agnostic Federated Learningcited by 456
  28. 2017VisDA: The Visual Domain Adaptation Challengecited by 435
  29. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  30. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
  31. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  32. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  33. 2020Adaptive Personalized Federated Learningcited by 358
  34. 2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 341
  35. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  36. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  37. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  38. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 246

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 6; 2024: 10; 2025: 9; 2026: 6.

36 papers.

Sort:

By area:

  1. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  2. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  3. 2022Topology-aware Generalization of Decentralized SGDcited by 38
  4. 2024Towards Efficient Replay in Federated Incremental Learningcited by 33
  5. 2023Integration of Large Language Models and Federated Learningcited by 31
  6. 2024Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learningcited by 23
  7. 2024Unleashing the Power of Continual Learning on Non-Centralized Devices: A Surveycited by 23
  8. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  9. 2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
  10. 2022Spatial-Temporal Federated Learning for Lifelong Person Re-identification on Distributed Edgescited by 12
  11. 2022Non-readily identifiable data collaboration analysis for multiple datasets including personal informationcited by 10
  12. 2024FedProK: Trustworthy Federated Class-Incremental Learning via Prototypical Feature Knowledge Transfercited by 9
24 more
  1. 2024Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Samplingcited by 7
  2. 2022Federated Continual Learning for Socially Aware Roboticscited by 5
  3. 2025A Thorough Assessment of the Non-IID Data Impact in Federated Learningcited by 5
  4. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  5. 2025AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Datacited by 3
  6. 2025Federated Learning Survey: A Multi-Level Taxonomy of Aggregation Techniques, Experimental Insights, and Future Frontierscited by 3
  7. 2022Decentralized digital twins of complex dynamical systemscited by 2
  8. 2025An Information-Theoretic Analysis for Federated Learning under Concept Driftcited by 2
  9. 2023Masked Autoencoders are Efficient Continual Federated Learnerscited by 1
  10. 2023Fingerprint Attack: Client De-Anonymization in Federated Learningcited by 1
  11. 2024Estimation of conditional average treatment effects on distributed confidential datacited by 1
  12. 2024Variational Bayes for Federated Continual Learningcited by 1
  13. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classificationcited by 1
  14. 2025FedTeddi: Temporal Drift and Divergence Aware Scheduling for Timely Federated Edge Learningcited by 1
  15. 2026Federated continual learning: A comprehensive survey on lifelong and privacy-preserving learning over distributed and non-stationary datacited by 1
  16. 2026C2FL: Clustered Continual Federated Learning under Spatial and Temporal Driftcited by 1
  17. 2023Global Layers: Non-IID Tabular Federated Learningcited by 0
  18. 2024Towards Privacy-Aware and Personalised Assistive Robots: A User-Centred Approachcited by 0
  19. 2025Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for FCLcited by 0
  20. 2025Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learningcited by 0
  21. 2026Agentic Federated Learning: The Future of Distributed Training Orchestrationcited by 0
  22. 2026Beyond Forgetting in Continual Medical Image Segmentation: A Comprehensive Benchmark Studycited by 0
  23. 2026A Comparative Study of Federated Learning Aggregation Strategies under Homogeneous and Heterogeneous Data Distributionscited by 0
  24. 2026PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Taskscited by 0