mlsanityData to 14 Oct 2026

Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search

arXiv 2605.21322 ↗First posted 20 May 2026cs.LGcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Federated Learning (FL) enables collaborative model training without centralizing data. However, real-world deployments must simultaneously address statistical heterogeneity across client data (non-IID), system heterogeneity in device capabilities, and communication efficiency. Existing FL approaches mitigate these challenges through improved aggregation, personalization, or knowledge distillation, but they almost universally assume a fixed client architecture, limiting adaptability to heterogeneous data complexity and hardware constraints. This architectural constraint often leads to suboptimal trade-offs between accuracy and efficiency in real-world FL systems. This work introduces FedKDNAS, a distillation-driven FL framework that combines client-side neural architecture selection with distillation of server-coordinated knowledge. Each client autonomously selects a lightweight model under accuracy-resource constraints. It then trains it locally using a hybrid objective combining supervised learning and knowledge distillation and shares only predictions on a public reference set. The server then aggregates and smooths these predictions, optionally combining them with a teacher model, to produce stable distillation targets for the next round. Extensive evaluation on six datasets against six representative FL baselines (FedAvg, Ditto, FedMD, FedDF, FedDistill, Local-KD) demonstrates that FedKDNAS consistently achieves superior Pareto efficiency, improving accuracy by up to 15% under non-IID conditions, reducing client CPU usage by approximately 28%, and decreasing communication overhead by up to 44 times while maintaining lightweight logit-based communication.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 44 of its 67 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

44 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning17 Mar
  2. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 42

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
30 more
  1. 2018Federated Learning with Non-IID Datacited by 1,108
  2. 2020Adaptive Federated Optimizationcited by 1,046
  3. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  4. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  7. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  8. 2019Green AIcited by 627
  9. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  10. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  11. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  12. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  13. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  14. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  15. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  16. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  17. 2018Large scale distributed neural network training through online distillationcited by 146
  18. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  19. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  20. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  21. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  22. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  23. 2020Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Searchcited by 30
  24. 2022What Knowledge Gets Distilled in Knowledge Distillation?cited by 27
  25. 2023FedET: A Communication-Efficient Federated Class-Incremental Learning Framework Based on Enhanced Transformercited by 23
  26. 2021SPIDER: Searching Personalized Neural Architecture for Federated Learningcited by 13
  27. 2019The State of Knowledge Distillation for Classificationcited by 9
  28. 2024FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learningcited by 4
  29. 2025FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learningcited by 2
  30. 2025One-Shot Neural Architecture Search with Network Similarity Directed Initialization for Pathological Image Classificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Training Heterogeneous Client Models using Knowledge Distillation in Serverless Federated Learning
  2. 2024Federated Distillation: A Survey
  3. 2023FedAL: Black-Box Federated Knowledge Distillation Enabled by Adversarial Learning
  4. 2022Closing the Gap between Client and Global Model Performance in Heterogeneous Federated Learning
  5. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillation
  6. 2026FedMTFI: Feature Importance Based Optimized Multi Teacher Knowledge Distillation in Heterogeneous Federated Learning Environment
  7. 2023Federated Learning on Non-iid Data via Local and Global Distillation
  8. 2022Knowledge Distillation for Federated Learning: a Practical Guide