Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
On the map: Knowledge distillation → Federated, robust and private distillation
Abstract (arXiv)
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. Data heterogeneity is one of the main challenges in FL, which results in slow convergence and degraded performance. Most existing approaches only tackle the heterogeneity challenge by restricting the local model update in client, ignoring the performance drop caused by direct global model aggregation. Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation. Concretely, FedFTG explores the input space of local models through a generator, and uses it to transfer the knowledge from local models to the global model. Besides, we propose a hard sample mining scheme to achieve effective knowledge distillation throughout the training. In addition, we develop customized label sampling and class-level ensemble to derive maximum utilization of knowledge, which implicitly mitigates the distribution discrepancy across clients. Extensive experiments show that our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
Key facts. Cited by 172 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 36 of its 45 references that are in the map. First posted on arXiv 17 Mar 2022.
Builds on
36 of 45 references.
Closest ancestors · 1
Same direction, newest first
Other references · 35
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
- 2021Model-Contrastive Federated Learningcited by 766
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
- 2018Federated Learning for Mobile Keyboard Predictioncited by 572
- 2021Federated Learning Based on Dynamic Regularizationcited by 562
- 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
23 more
- 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
- 2021A Field Guide to Federated Optimizationcited by 315
- 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
- 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 287
- 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
- 2019Model Fusion via Optimal Transportcited by 219
- 2019Data-Free Learning of Student Networkscited by 204
- 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
- 2015A Large-Scale Car Dataset for Fine-Grained Categorization and Verificationcited by 130
- 2020Optimal Client Sampling for Federated Learningcited by 124
- 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
- 2021Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learningcited by 93
- 2019Data-Free Adversarial Distillationcited by 90
- 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
- 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
- 2020Performance Optimization for Federated Person Re-identification via Benchmark Analysiscited by 50
- 2020Stochastic Client Selection for Federated Learning with Volatile Clientscited by 39
- 2019Private Federated Learning with Domain Adaptationcited by 36
- 2019Federated Learning for Ranking Browser History Suggestionscited by 24
- 2020Particle Swarm Optimized Federated Learning For Industrial IoT and Smart City Servicescited by 18
- 2020Decentralised Learning from Independent Multi-Domain Labels for Person Re-Identificationcited by 16
- 2019HHHFL: Hierarchical Heterogeneous Horizontal Federated Learning for Electroencephalographycited by 15
- 2019Learn Electronic Health Records by Fully Decentralized Federated Learningcited by 5
Built on by
Citing papers per year: 2022: 16; 2023: 58; 2024: 46; 2025: 33; 2026: 19.
172 papers.
Sort:
By area:
- 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
- 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
- 2023Deep Model Fusion: A Surveycited by 91
- 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
- 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
- 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
- 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
- 2023Fair Federated Medical Image Segmentation via Client Contribution Estimationcited by 52
- 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
- 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
- 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
38 more
- 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
- 2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
- 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
- 2024Towards Efficient Replay in Federated Incremental Learningcited by 33
- 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
- 2023Towards Stable Backdoor Purification through Feature Shift Tuningcited by 32
- 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
- 2022FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learningcited by 27
- 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
- 2024Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boostingcited by 27
- 2024FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learningcited by 27
- 2022Meta Knowledge Condensation for Federated Learningcited by 26
- 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
- 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
- 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
- 2023Federated Domain Generalization: A Surveycited by 25
- 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
- 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
- 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
- 2022DYNAFED: Tackling Client Data Heterogeneity with Global Dynamicscited by 22
- 2022FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregationcited by 21
- 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
- 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
- 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
- 2023Learning to Learn from APIs: Black-Box Data-Free Meta-Learningcited by 18
- 2024An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learningcited by 17
- 2023Revisiting Personalized Federated Learning: Robustness Against Backdoor Attackscited by 16
- 2023PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guaranteescited by 15
- 2024Federated Distillation: A Surveycited by 15
- 2024Threats and Defenses in Federated Learning Life Cycle: A Comprehensive Survey and Challengescited by 15
- 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
- 2023Exploiting Label Skews in Federated Learning with Model Concatenationcited by 14
- 2023Feature Matching Data Synthesis for Non-IID Federated Learningcited by 13
- 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
- 2023Federated Learning over Coupled Graphscited by 12
- 2023FedACK: Federated Adversarial Contrastive Knowledge Distillation for Cross-Lingual and Cross-Model Social Bot Detectioncited by 12
- 2023FedCiR: Client-Invariant Representation Learning for Federated Non-IID Featurescited by 12
- 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
50 most cited of 172 All 172 (JSON)
- 2026Joint Optimization for Federated Learning and Transmission over Unreliable Wireless Networks with Heterogeneous Data13 Sep 2026
- 2026Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network1 Sep 2026
- 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxy9 Aug 2026
- 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning15 Jul 2026
- 2026Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning8 Jul 2026
- 2026FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learning30 Jun 2026
- 2026GEN-Guard: Correcting Generalization Failures for Deployable Federated Surgical AI18 Jun 2026
- 2026Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning8 Jun 2026
- 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search20 May 2026
- 2026Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs11 May 2026
- 2026Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective10 May 2026
- 2026A Taxonomy and Resolution Strategy for Client-Level Disagreements in Federated Learning25 Apr 2026
38 more
- 2026FedRio: Personalized Federated Social Bot Detection via Cooperative Reinforced Contrastive Adversarial Distillation12 Apr 2026
- 2026Collaborative Adaptive Curriculum for Progressive Knowledge Distillation19 Mar 2026
- 2026FedCova: Robust Federated Covariance Learning Against Noisy Labels4 Mar 2026
- 2026FedDAG: Clustered Federated Learning via Global Data and Gradient Integration for Heterogeneous Environments26 Feb 2026
- 2026GFPL: Generative Federated Prototype Learning for Resource-Constrained and Data-Imbalanced Vision Task25 Feb 2026
- 2026The Gaussian-Head OFL Family: One-Shot Federated Learning from Client Global Statistics1 Feb 2026
- 2026Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detection1 Feb 2026
- 2025Mobility-Assisted Decentralized Federated Learning: Convergence Analysis and A Data-Driven Approach31 Dec 2025
- 2025Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learning25 Nov 2025
- 2025Dynamic Participation in Federated Learning: Benchmarks and a Knowledge Pool Plugin20 Nov 2025
- 2025SAM-Fed: SAM-Guided Federated Semi-Supervised Learning for Medical Image Segmentation18 Nov 2025
- 2025Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning1 Nov 2025
- 2025Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications31 Oct 2025
- 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning23 Oct 2025
- 2025Cooperative Pseudo Labeling for Unsupervised Federated Classification11 Oct 2025
- 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights5 Sep 2025
- 2025Dual-Distilled Heterogeneous Federated Learning with Adaptive Margins for Trainable Global Prototypes26 Aug 2025
- 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning14 Jul 2025
- 2025When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need5 Jul 2025
- 2025FedSC: Federated Learning with Semantic-Aware Collaboration26 Jun 2025
- 2025Efficient Federated Learning with Encrypted Data Sharing for Data-Heterogeneous Edge Devices25 Jun 2025
- 2025Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation25 Jun 2025
- 2025Distilling A Universal Expert from Clustered Federated Learning25 Jun 2025
- 2025UniVarFL: Uniformity and Variance Regularized Federated Learning for Heterogeneous Data9 Jun 2025
- 2025Enhancing Federated Class-Incremental Learning via Spatial-Temporal Statistics Aggregation2 Jun 2025
- 2025Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments26 May 2025
- 2025FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaboration25 May 2025
- 2025Redefining Clustered Federated Learning for System Identification: The Path of ClusterCraft22 May 2025
- 2025Optimal Client Sampling in Federated Learning with Client-Level Heterogeneous Differential Privacy19 May 2025
- 2025Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible5 Apr 2025
- 2025Adversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks1 Apr 2025
- 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning14 Mar 2025
- 2025You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Data10 Mar 2025
- 2025Federated Learning for Diffusion Models9 Mar 2025
- 2025Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation8 Mar 2025
- 2025Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data27 Feb 2025
- 2025FedBM: Stealing Knowledge from Pre-trained Language Models for Heterogeneous Federated Learning24 Feb 2025
- 2025FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devices12 Feb 2025
50 newest of 172 All 172 (JSON)
- 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
- 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
- 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
- 2024FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learningcited by 27
- 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
- 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
- 2023Learning to Learn from APIs: Black-Box Data-Free Meta-Learningcited by 18
- 2023FedACK: Federated Adversarial Contrastive Knowledge Distillation for Cross-Lingual and Cross-Model Social Bot Detectioncited by 12
- 2024Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillationcited by 12
- 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 9
- 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
- 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
12 most cited of 27 All 27 (JSON)
- 2026Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network1 Sep 2026
- 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search20 May 2026
- 2026Collaborative Adaptive Curriculum for Progressive Knowledge Distillation19 Mar 2026
- 2025Reviving Stale Updates: Data-Free Knowledge Distillation for Asynchronous Federated Learning1 Nov 2025
- 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning14 Jul 2025
- 2025When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need5 Jul 2025
- 2025Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation25 Jun 2025
- 2025Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible5 Apr 2025
- 2025Adversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks1 Apr 2025
- 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learning14 Mar 2025
- 2024HYDRA-FL: Hybrid Knowledge Distillation for Robust and Accurate Federated Learning30 Sep 2024
- 2024Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator11 Sep 2024
12 newest of 27 All 27 (JSON)
- 2023Revisiting Data-Free Knowledge Distillation with Poisoned Teacherscited by 9
- 2024Dataset Size Recovery from LoRA Weightscited by 7
- 2024Dataset Size Recovery from LoRA Weights27 Jun 2024
- 2023Revisiting Data-Free Knowledge Distillation with Poisoned Teachers4 Jun 2023
- 2025Federated Learning for Diffusion Modelscited by 7
- 2025Federated Learning for Diffusion Models9 Mar 2025
- 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
- 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
- 2023Deep Model Fusion: A Surveycited by 91
- 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
- 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
- 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
- 2023Fair Federated Medical Image Segmentation via Client Contribution Estimationcited by 52
- 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
- 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
- 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
- 2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
12 most cited of 140 All 140 (JSON)
- 2026Joint Optimization for Federated Learning and Transmission over Unreliable Wireless Networks with Heterogeneous Data13 Sep 2026
- 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxy9 Aug 2026
- 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning15 Jul 2026
- 2026Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning8 Jul 2026
- 2026FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learning30 Jun 2026
- 2026GEN-Guard: Correcting Generalization Failures for Deployable Federated Surgical AI18 Jun 2026
- 2026Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning8 Jun 2026
- 2026Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs11 May 2026
- 2026Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective10 May 2026
- 2026A Taxonomy and Resolution Strategy for Client-Level Disagreements in Federated Learning25 Apr 2026
- 2026FedRio: Personalized Federated Social Bot Detection via Cooperative Reinforced Contrastive Adversarial Distillation12 Apr 2026
- 2026FedCova: Robust Federated Covariance Learning Against Noisy Labels4 Mar 2026
12 newest of 140 All 140 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2022Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
- 2023Federated Learning on Non-iid Data via Local and Global Distillation
- 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
- 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data
- 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
- 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
- 2024Tailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillation
- 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillation