Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data
Not on the map. Why?
Abstract (arXiv)
The widespread success of pre-trained language models has established a new training paradigm, where a global PLM is fine-tuned using task-specific data from local clients. The local data are highly different from each other and can not capture the global distribution of the whole data in real world. To address the challenges of non-IID data in real environments, privacy-preserving federated distillation has been proposed and highly investigated. However, previous experimental non-IID scenarios are primarily identified with the label (output) diversity, without considering the diversity of language domains (input) that is crucial in natural language processing. In this paper, we introduce a comprehensive set of multi-domain non-IID scenarios and propose a unified benchmarking framework that includes diverse data. The benchmark can be used to evaluate the federated learning framework in a real environment. To this end, we propose an Adaptive Federated Distillation (AdaFD) framework designed to address multi-domain non-IID challenges in both homogeneous and heterogeneous settings. Experimental results demonstrate that our models capture the diversity of local clients and achieve better performance compared to the existing works. The code for this paper is available at: https://github.com/jiahaoxiao1228/AdaFD.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 46 references that are in the map. First posted on arXiv 28 Aug 2025.
Builds on
24 of 46 references.
References · 24
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
- 2021Model-Contrastive Federated Learningcited by 766
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
- 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
- 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
- 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
- 2021Federated Learning on Non-IID Data: A Surveycited by 346
- 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
- 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
12 more
- 2022Vertical Federated Learning: Concepts, Advances and Challengescited by 108
- 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
- 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
- 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
- 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
- 2021Federated Learning Meets Natural Language Processing: A Surveycited by 43
- 2022Faster Adaptive Federated Learningcited by 35
- 2020FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learningcited by 24
- 2023Feature Matching Data Synthesis for Non-IID Federated Learningcited by 13
- 2022FedDKD: Federated Learning with Decentralized Knowledge Distillationcited by 9
- 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learningcited by 5
- 2022Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learningcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.