# Federated Learning with Extremely Noisy Clients via Negative Distillation

arXiv: https://arxiv.org/abs/2312.12703 · HTML: https://mlsanity.com/paper/2312.12703/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 23 of 23)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Masking: A New Perspective of Noisy Supervision](https://mlsanity.com/paper/1805.08193/) (2018)
- [Jo-SRC: A Contrastive Approach for Combating Noisy Labels](https://mlsanity.com/paper/2103.13029/) (2021)
- [Tackling Data Heterogeneity in Federated Learning with Class Prototypes](https://mlsanity.com/paper/2212.02758/) (2022)
- [FedCorr: Multi-Stage Federated Learning for Label Noise Correction](https://mlsanity.com/paper/2204.04677/) (2022)
- [FedFed: Feature Distillation against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2310.05077/) (2023)
- [Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2206.02465/) (2022)
- [FOCUS: Dealing with Label Quality Disparity in Federated Learning](https://mlsanity.com/paper/2001.11359/) (2020)
- [FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneity](https://mlsanity.com/paper/2305.05230/) (2023)
- [FedNoisy: Federated Noisy Label Learning Benchmark](https://mlsanity.com/paper/2306.11650/) (2023)
- [FedRN: Exploiting k-Reliable Neighbors Towards Robust Federated Learning](https://mlsanity.com/paper/2205.01310/) (2022)

## Built on by (most cited first, 12 of 12)

- [FNBench: Benchmarking Robust Federated Learning against Noisy Labels](https://mlsanity.com/paper/2505.06684/) (2025)
- [FedGreed: A Byzantine-Robust Loss-Based Aggregation Method for Federated Learning](https://mlsanity.com/paper/2508.18060/) (2025)
- [Robust Federated Learning under Adversarial Attacks via Loss-Based Client Clustering](https://mlsanity.com/paper/2508.12672/) (2025)
- [FB-NLL: A Feature-Based Approach to Tackle Noisy Labels in Personalized Federated Learning](https://mlsanity.com/paper/2604.19729/) (2026)
- [Beyond Aggregation: Guiding Clients in Heterogeneous Federated Learning](https://mlsanity.com/paper/2509.23049/) (2025)
- [Can we Trust Unreliable Voxels? Exploring 3D Semantic Occupancy Prediction under Label Noise](https://mlsanity.com/paper/2603.06279/) (2026)
- [FedSIR: Spectral Client Identification and Relabeling for Federated Learning with Noisy Labels](https://mlsanity.com/paper/2604.20825/) (2026)
- [Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels](https://mlsanity.com/paper/2412.00452/) (2024)
- [Local K-Similarity Constraint for Federated Learning with Label Noise](https://mlsanity.com/paper/2511.06169/) (2025)
- [FedCova: Robust Federated Covariance Learning Against Noisy Labels](https://mlsanity.com/paper/2603.04062/) (2026)
- [OCT-FedSIR: Toward Trustworthy Federated Ophthalmic Learning under Annotation Noise](https://mlsanity.com/paper/2609.14734/) (2026)
- [Different Corruptions, Different Signals: Uncertainty and Loss in Federated Data Quality](https://mlsanity.com/paper/2609.31454/) (2026)

## Abstract (arXiv)

Federated learning (FL) has shown remarkable success in cooperatively training deep models, while typically struggling with noisy labels. Advanced works propose to tackle label noise by a re-weighting strategy with a strong assumption, i.e., mild label noise. However, it may be violated in many real-world FL scenarios because of highly contaminated clients, resulting in extreme noise ratios, e.g., >90%. To tackle extremely noisy clients, we study the robustness of the re-weighting strategy, showing a pessimistic conclusion: minimizing the weight of clients trained over noisy data outperforms re-weighting strategies. To leverage models trained on noisy clients, we propose a novel approach, called negative distillation (FedNed). FedNed first identifies noisy clients and employs rather than discards the noisy clients in a knowledge distillation manner. In particular, clients identified as noisy ones are required to train models using noisy labels and pseudo-labels obtained by global models. The model trained on noisy labels serves as a `bad teacher' in knowledge distillation, aiming to decrease the risk of providing incorrect information. Meanwhile, the model trained on pseudo-labels is involved in model aggregation if not identified as a noisy client. Consequently, through pseudo-labeling, FedNed gradually increases the trustworthiness of models trained on noisy clients, while leveraging all clients for model aggregation through negative distillation. To verify the efficacy of FedNed, we conduct extensive experiments under various settings, demonstrating that FedNed can consistently outperform baselines and achieve state-of-the-art performance. Our code is available at https://github.com/linChen99/FedNed.