# Federated Learning with Label-Masking Distillation

arXiv: https://arxiv.org/abs/2409.13136 · HTML: https://mlsanity.com/paper/2409.13136/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 28 of 28)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [Federated Learning for Smart Healthcare: A Survey](https://mlsanity.com/paper/2111.08834/) (2021)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Privacy-preserving Federated Brain Tumour Segmentation](https://mlsanity.com/paper/1910.00962/) (2019)
- [NLNL: Negative Learning for Noisy Labels](https://mlsanity.com/paper/1908.07387/) (2019)
- [Overcoming Forgetting in Federated Learning on Non-IID Data](https://mlsanity.com/paper/1910.07796/) (2019)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [Federated Learning with Only Positive Labels](https://mlsanity.com/paper/2004.10342/) (2020)
- [Performance Optimization for Federated Person Re-identification via Benchmark Analysis](https://mlsanity.com/paper/2008.11560/) (2020)
- [Communication Efficiency in Federated Learning: Achievements and Challenges](https://mlsanity.com/paper/2107.10996/) (2021)
- [Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data](https://mlsanity.com/paper/2107.00051/) (2021)
- [Joint Optimization in Edge-Cloud Continuum for Federated Unsupervised Person Re-identification](https://mlsanity.com/paper/2108.06493/) (2021)
- [Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space](https://mlsanity.com/paper/2104.00352/) (2021)

## Built on by (most cited first, 7 of 7)

- [Dataset Distillation-based Hybrid Federated Learning on Non-IID Data](https://mlsanity.com/paper/2409.17517/) (2024)
- [Federated Hybrid Training and Self-Adversarial Distillation: Towards Robust Edge Networks](https://mlsanity.com/paper/2412.19354/) (2024)
- [Federated Progressive Self-Distillation with Logits Calibration for Personalized IIoT Edge Intelligence](https://mlsanity.com/paper/2412.00410/) (2024)
- [SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2507.08508/) (2025)
- [FedADB: Class Anchor-Driven Dual-Branch Federated Learning for Mitigating Forgetting](https://mlsanity.com/paper/2608.15310/) (2026)
- [FedCC: Towards Addressing Label Distribution Skews in Distillation-Based Federated Learning](https://mlsanity.com/paper/2608.23031/) (2026)
- [Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detection](https://mlsanity.com/paper/2609.31785/) (2026)

## Abstract (arXiv)

Federated learning provides a privacy-preserving manner to collaboratively train models on data distributed over multiple local clients via the coordination of a global server. In this paper, we focus on label distribution skew in federated learning, where due to the different user behavior of the client, label distributions between different clients are significantly different. When faced with such cases, most existing methods will lead to a suboptimal optimization due to the inadequate utilization of label distribution information in clients. Inspired by this, we propose a label-masking distillation approach termed FedLMD to facilitate federated learning via perceiving the various label distributions of each client. We classify the labels into majority and minority labels based on the number of examples per class during training. The client model learns the knowledge of majority labels from local data. The process of distillation masks out the predictions of majority labels from the global model, so that it can focus more on preserving the minority label knowledge of the client. A series of experiments show that the proposed approach can achieve state-of-the-art performance in various cases. Moreover, considering the limited resources of the clients, we propose a variant FedLMD-Tf that does not require an additional teacher, which outperforms previous lightweight approaches without increasing computational costs. Our code is available at https://github.com/wnma3mz/FedLMD.