# MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare

arXiv: https://arxiv.org/abs/2206.08516 · HTML: https://mlsanity.com/paper/2206.08516/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 70 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 28 of 28)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](https://mlsanity.com/paper/1907.09173/) (2019)
- [Salvaging Federated Learning by Local Adaptation](https://mlsanity.com/paper/2002.04758/) (2020)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning](https://mlsanity.com/paper/1905.06731/) (2019)
- [Medical Image Segmentation Using Squeeze-and-Expansion Transformers](https://mlsanity.com/paper/2105.09511/) (2021)
- [A distillation-based approach integrating continual learning and federated learning for pervasive services](https://mlsanity.com/paper/2109.04197/) (2021)
- [Towards Model Agnostic Federated Learning Using Knowledge Distillation](https://mlsanity.com/paper/2110.15210/) (2021)
- [Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition](https://mlsanity.com/paper/2206.06629/) (2022)
- [Personalized Federated Learning with Adaptive Batchnorm for Healthcare](https://mlsanity.com/paper/2112.00734/) (2021)
- [Cross-domain Activity Recognition via Substructural Optimal Transport](https://mlsanity.com/paper/2102.03353/) (2021)
- [FedFMC: Sequential Efficient Federated Learning on Non-iid Data](https://mlsanity.com/paper/2006.10937/) (2020)
- [FedH2L: Federated Learning with Model and Statistical Heterogeneity](https://mlsanity.com/paper/2101.11296/) (2021)
- [Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training](https://mlsanity.com/paper/2201.12976/) (2022)
- [Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients](https://mlsanity.com/paper/2201.10899/) (2022)
- [Locality-aware Channel-wise Dropout for Occluded Face Recognition](https://mlsanity.com/paper/2107.09270/) (2021)
- [DeepKeyGen: A Deep Learning-based Stream Cipher Generator for Medical Image Encryption and Decryption](https://mlsanity.com/paper/2012.11097/) (2020)

## Built on by (most cited first, 30 of 37)

- [FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning](https://mlsanity.com/paper/2302.13485/) (2023)
- [Eliminating Domain Bias for Federated Learning in Representation Space](https://mlsanity.com/paper/2311.14975/) (2023)
- [GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learning](https://mlsanity.com/paper/2308.10279/) (2023)
- [Multimodal Federated Learning in Healthcare: a Review](https://mlsanity.com/paper/2310.09650/) (2023)
- [Beyond the Hype: A dispassionate look at vision-language models in medical scenario](https://mlsanity.com/paper/2408.08704/) (2024)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [One-Shot Sequential Federated Learning for Non-IID Data by Enhancing Local Model Diversity](https://mlsanity.com/paper/2404.12130/) (2024)
- [Advancements in Federated Learning: Models, Methods, and Privacy](https://mlsanity.com/paper/2302.11466/) (2023)
- [FedLPPA: Learning Personalized Prompt and Aggregation for Federated Weakly-supervised Medical Image Segmentation](https://mlsanity.com/paper/2402.17502/) (2024)
- [Knowledge Distillation in Federated Edge Learning: A Survey](https://mlsanity.com/paper/2301.05849/) (2023)
- [Unifying and Personalizing Weakly-supervised Federated Medical Image Segmentation via Adaptive Representation and Aggregation](https://mlsanity.com/paper/2304.05635/) (2023)
- [Rethinking Client Drift in Federated Learning: A Logit Perspective](https://mlsanity.com/paper/2308.10162/) (2023)
- [RobWE: Robust Watermark Embedding for Personalized Federated Learning Model Ownership Protection](https://mlsanity.com/paper/2402.19054/) (2024)
- [F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics](https://mlsanity.com/paper/2411.11912/) (2024)
- [Data-Free Federated Class Incremental Learning with Diffusion-Based Generative Memory](https://mlsanity.com/paper/2405.17457/) (2024)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [Towards Optimal Customized Architecture for Heterogeneous Federated Learning with Contrastive Cloud-Edge Model Decoupling](https://mlsanity.com/paper/2403.02360/) (2024)
- [Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalization](https://mlsanity.com/paper/2408.07966/) (2024)
- [MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks](https://mlsanity.com/paper/2502.01158/) (2025)
- [Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients](https://mlsanity.com/paper/2310.07171/) (2023)
- [Fairness-Aware Job Scheduling for Multi-Job Federated Learning](https://mlsanity.com/paper/2401.02740/) (2024)
- [Multiple Teachers-Meticulous Student: A Domain Adaptive Meta-Knowledge Distillation Model for Medical Image Classification](https://mlsanity.com/paper/2403.11226/) (2024)
- [KnFu: Effective Knowledge Fusion](https://mlsanity.com/paper/2403.11892/) (2024)
- [FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning](https://mlsanity.com/paper/2402.09095/) (2024)
- [FedBrain-Distill: Communication-Efficient Federated Brain Tumor Classification Using Ensemble Knowledge Distillation on Non-IID Data](https://mlsanity.com/paper/2409.05359/) (2024)
- [Certifiably Robust Model Evaluation in Federated Learning under Meta-Distributional Shifts](https://mlsanity.com/paper/2410.20250/) (2024)
- [Task-Agnostic Federation over Decentralized Data: Research Landscape and Visions](https://mlsanity.com/paper/2503.03140/) (2025)
- [A Linearized Alternating Direction Multiplier Method for Federated Matrix Completion Problems](https://mlsanity.com/paper/2503.12733/) (2025)
- [FHBench: Towards Efficient and Personalized Federated Learning for Multimodal Healthcare](https://mlsanity.com/paper/2504.10817/) (2025)
- [FedX: Adaptive Model Decomposition and Quantization for IoT Federated Learning](https://mlsanity.com/paper/2504.12849/) (2025)

## Abstract (arXiv)

Federated learning has attracted increasing attention to building models without accessing the raw user data, especially in healthcare. In real applications, different federations can seldom work together due to possible reasons such as data heterogeneity and distrust/inexistence of the central server. In this paper, we propose a novel framework called MetaFed to facilitate trustworthy FL between different federations. MetaFed obtains a personalized model for each federation without a central server via the proposed Cyclic Knowledge Distillation. Specifically, MetaFed treats each federation as a meta distribution and aggregates knowledge of each federation in a cyclic manner. The training is split into two parts: common knowledge accumulation and personalization. Comprehensive experiments on three benchmarks demonstrate that MetaFed without a server achieves better accuracy compared to state-of-the-art methods (e.g., 10%+ accuracy improvement compared to the baseline for PAMAP2) with fewer communication costs.