mlsanityData to 7 Oct 2026

Preservation of the Global Knowledge by Not-True Distillation in Federated Learning

arXiv 2106.03097 ↗First posted 6 Jun 2021cs.LG cs.AI cs.CVcited by 87

Not on the map. Why?

Abstract (arXiv)

In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that tackling down forgetting will relieve the data heterogeneity problem. To this end, we propose a novel and effective algorithm, Federated Not-True Distillation (FedNTD), which preserves the global perspective on locally available data only for the not-true classes. In the experiments, FedNTD shows state-of-the-art performance on various setups without compromising data privacy or incurring additional communication costs.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 40 of its 66 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

40 of 66 references.

References · 40

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2016Learning without Forgettingcited by 2,465
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
28 more
  1. 2018Federated Learning with Non-IID Datacited by 1,099
  2. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  5. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  6. 2021Model-Contrastive Federated Learningcited by 762
  7. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  8. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  11. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  12. 2020Federated Learning with Matched Averagingcited by 564
  13. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  14. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  15. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  16. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  17. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  18. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  19. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  20. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  21. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  22. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 160
  23. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 158
  24. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  25. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  26. 2020Distilled One-Shot Federated Learningcited by 108
  27. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  28. 2020CPR: Classifier-Projection Regularization for Continual Learningcited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 19; 2024: 30; 2025: 27; 2026: 6.

87 papers.

Sort:

By area:

  1. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  2. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  3. 2022FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reductioncited by 35
  4. 2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
  5. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  6. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  7. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  8. 2024Federated Unlearning: A Survey on Methods, Design Guidelines, and Evaluation Metricscited by 25
  9. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  10. 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
  11. 2023Federated Generative Learning with Foundation Modelscited by 23
  12. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
38 more
  1. 2023A Simple Data Augmentation for Feature Distribution Skewed Federated Learningcited by 16
  2. 2023A Survey for Federated Learning Evaluations: Goals and Measurescited by 16
  3. 2023PFLlib: A Beginner-Friendly and Comprehensive Personalized Federated Learning Library and Benchmarkcited by 16
  4. 2024Federated Distillation: A Surveycited by 14
  5. 2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
  6. 2024AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Modelscited by 11
  7. 2025Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learningcited by 11
  8. 2024Exploring Vacant Classes in Label-Skewed Federated Learningcited by 10
  9. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillationcited by 9
  10. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  11. 2023Towards Instance-adaptive Inference for Federated Learningcited by 7
  12. 2025S2FGL: Spatial Spectral Federated Graph Learningcited by 7
  13. 2023Continual Adaptation of Vision Transformers for Federated Learningcited by 6
  14. 2023Dynamic Heterogeneous Federated Learning with Multi-Level Prototypescited by 6
  15. 2024Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuningcited by 6
  16. 2025Subgraph Federated Learning for Local Generalizationcited by 6
  17. 2023Knowledge Distillation in Federated Edge Learning: A Surveycited by 5
  18. 2023Mitigating Group Bias in Federated Learning for Heterogeneous Devicescited by 5
  19. 2023Federated Skewed Label Learning with Logits Fusioncited by 5
  20. 2024Lightweight Unsupervised Federated Learning with Pretrained Vision Language Modelcited by 5
  21. 2024Federated Class-Incremental Learning with New-Class Augmented Self-Distillationcited by 4
  22. 2024Flashback: Understanding and Mitigating Forgetting in Federated Learningcited by 4
  23. 2024Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspectivecited by 4
  24. 2025Permutation-Invariant Representation Learning for Robust and Privacy-Preserving Feature Selectioncited by 4
  25. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  26. 2023FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learningcited by 3
  27. 2023Towards Fast and Stable Federated Learning: Confronting Heterogeneity via Knowledge Anchorcited by 3
  28. 2024Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Modelscited by 3
  29. 2024FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learningcited by 3
  30. 2024The Diversity Bonus: Learning from Dissimilar Distributed Clients in Personalized Federated Learningcited by 3
  31. 2025Understanding Federated Learning from IID to Non-IID dataset: An Experimental Studycited by 3
  32. 2025AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Datacited by 3
  33. 2025FedSC: Federated Learning with Semantic-Aware Collaborationcited by 3
  34. 2024SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustmentscited by 2
  35. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  36. 2024FedQUIT: On-Device Federated Unlearning via a Quasi-Competent Virtual Teachercited by 2
  37. 2024Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalizationcited by 2
  38. 2024Dataset Distillation-based Hybrid Federated Learning on Non-IID Datacited by 2

50 most cited of 87 All 87 (JSON)