mlsanityData to 14 Oct 2026

Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning

arXiv 2511.00655 ↗First posted 1 Nov 2025cs.LGcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, yet its scalability is limited by synchronization overhead. Asynchronous federated learning (AFL) alleviates this issue by allowing clients to communicate independently, thereby improving wall-clock efficiency in large-scale, hardware-heterogeneous environments. However, asynchrony introduces updates computed on outdated global models (staleness) that can destabilize optimization and hinder convergence. We propose FedRevive, an AFL framework that revives stale updates through data-free knowledge distillation (DFKD). FedRevive integrates parameter-space aggregation with a server-side DFKD process that transfers knowledge from stale client updates to the current global model without access to data. A meta-learned generator synthesizes pseudo-samples for multi-teacher distillation. A hybrid aggregation scheme combining raw with DFKD updates effectively mitigates staleness while retaining AFL scalability. Experiments on vision and text benchmarks show that FedRevive achieves faster training by up to 29.7% and higher final accuracy by up to 11.4 percentage points than baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 46 of its 66 references that are in the map. First posted on arXiv 1 Nov 2025.

Builds on

46 of 66 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible5 Apr
  2. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions16 Jun
  3. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning17 Mar
  4. 2022FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks10 Jan
  5. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  10. 2019Towards Federated Learning at Scale: System Designcited by 832
  11. 2018LEAF: A Benchmark for Federated Settingscited by 792
  12. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
29 more
  1. 2020The Future of Digital Health with Federated Learningcited by 633
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  3. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2019Asynchronous Federated Optimizationcited by 267
  7. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  8. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  9. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  10. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  11. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 116
  12. 2019Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregationcited by 100
  13. 2021Asynchronous Federated Learning on Heterogeneous Devices: A Surveycited by 93
  14. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
  15. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  16. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  17. 2025A Comprehensive Survey on Knowledge Distillationcited by 51
  18. 2024Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillationcited by 47
  19. 2015Staleness-aware Async-SGD for Distributed Deep Learningcited by 43
  20. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  21. 2015Model Accuracy and Runtime Tradeoff in Distributed Deep Learning:A Systematic Studycited by 19
  22. 2024Federated Distillation: A Surveycited by 15
  23. 2024Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scalecited by 14
  24. 2023Async-HFL: Efficient and Robust Asynchronous Federated Learning in Hierarchical IoT Networkscited by 11
  25. 2021FedSLD: Federated Learning with Shared Label Distribution for Medical Image Classificationcited by 10
  26. 2023A Survey on Model-heterogeneous Federated Learning: Problems, Methods, and Prospectscited by 9
  27. 2024FedAST: Federated Asynchronous Simultaneous Trainingcited by 3
  28. 2024Overcoming label shift with target-aware federated learningcited by 2
  29. 2025C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Asynchronous Byzantine Federated Learning
  2. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  3. 2026DriftGuard: Mitigating Asynchronous Data Drift in Federated Learning
  4. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillation
  5. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning
  6. 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
  7. 2021FedHe: Heterogeneous Models and Communication-Efficient Federated Learning
  8. 2025Towards Collaborative Fairness in Federated Learning Under Imbalanced Covariate Shift