mlsanityData to 7 Oct 2026

CADIS: Handling Cluster-skewed Non-IID Data in Federated Learning with Clustered Aggregation and Knowledge DIStilled Regularization

arXiv 2302.10413 ↗First posted 21 Feb 2023cs.LG cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Federated learning enables edge devices to train a global model collaboratively without exposing their data. Despite achieving outstanding advantages in computing efficiency and privacy protection, federated learning faces a significant challenge when dealing with non-IID data, i.e., data generated by clients that are typically not independent and identically distributed. In this paper, we tackle a new type of Non-IID data, called cluster-skewed non-IID, discovered in actual data sets. The cluster-skewed non-IID is a phenomenon in which clients can be grouped into clusters with similar data distributions. By performing an in-depth analysis of the behavior of a classification model's penultimate layer, we introduce a metric that quantifies the similarity between two clients' data distributions without violating their privacy. We then propose an aggregation scheme that guarantees equality between clusters. In addition, we offer a novel local training regularization based on the knowledge-distillation technique that reduces the overfitting problem at clients and dramatically boosts the training scheme's performance. We theoretically prove the superiority of the proposed aggregation over the benchmark FedAvg. Extensive experimental results on both standard public datasets and our in-house real-world dataset demonstrate that the proposed approach improves accuracy by up to 16% compared to the FedAvg algorithm.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

23 of 51 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  7. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  8. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  9. 2019Agnostic Federated Learningcited by 451
  10. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  11. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  12. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
11 more
  1. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  2. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  3. 2020Fast-Convergent Federated Learning with Adaptive Weightingcited by 67
  4. 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
  5. 2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 62
  6. 2021Federated Learning with Fair Averagingcited by 59
  7. 2021Emerging Trends in Federated Learning: From Model Fusion to Federated X Learningcited by 35
  8. 2020Fairness and Accuracy in Federated Learningcited by 27
  9. 2020FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cyclingcited by 26
  10. 2021A Triangle Inequality for Cosine Similaritycited by 13
  11. 2022FedDRL: Deep Reinforcement Learning-based Adaptive Aggregation for Non-IID Data in Federated Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularizationcited by 16
  2. 2024Federated Distillation: A Surveycited by 14
  3. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  4. 2025Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Datacited by 7
  5. 2026Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Datacited by 3