mlsanityData to 14 Oct 2026

Query-based Knowledge Transfer for Heterogeneous Learning Environments

arXiv 2504.09205 ↗First posted 12 Apr 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Decentralized collaborative learning under data heterogeneity and privacy constraints has rapidly advanced. However, existing solutions like federated learning, ensembles, and transfer learning, often fail to adequately serve the unique needs of clients, especially when local data representation is limited. To address this issue, we propose a novel framework called Query-based Knowledge Transfer (QKT) that enables tailored knowledge acquisition to fulfill specific client needs without direct data exchange. QKT employs a data-free masking strategy to facilitate communication-efficient query-focused knowledge transfer while refining task-specific parameters to mitigate knowledge interference and forgetting. Our experiments, conducted on both standard and clinical benchmarks, show that QKT significantly outperforms existing collaborative learning methods by an average of 20.91% points in single-class query settings and an average of 14.32% points in multi-class query scenarios. Further analysis and ablation studies reveal that QKT effectively balances the learning of new and existing knowledge, showing strong potential for its application in decentralized learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 12 Apr 2025.

Builds on

32 of 48 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2016Learning without Forgettingcited by 2,496
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  11. 2021Model-Contrastive Federated Learningcited by 766
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
20 more
  1. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  2. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  3. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  4. 2020Federated Learning with Matched Averagingcited by 564
  5. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  6. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  7. 2021Towards Personalized Federated Learningcited by 464
  8. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  9. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  10. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  11. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  12. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 223
  13. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  14. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 67
  15. 2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
  16. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  17. 2022Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matchingcited by 24
  18. 2021Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learningcited by 23
  19. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 9
  20. 2024Practical Insights into Knowledge Distillation for Pre-Trained Modelscited by 2

Built on by

Not cited yet.