# FEDIC: Federated Learning on Non-IID and Long-Tailed Data via Calibrated Distillation

arXiv: https://arxiv.org/abs/2205.00172 · HTML: https://mlsanity.com/paper/2205.00172/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 30 Apr 2022.

## Builds on (most cited first, 17 of 17)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Distribution Alignment: A Unified Framework for Long-tail Visual Recognition](https://mlsanity.com/paper/2103.16370/) (2021)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [Remix: Rebalanced Mixup](https://mlsanity.com/paper/2007.03943/) (2020)
- [Fed-Focal Loss for imbalanced data classification in Federated Learning](https://mlsanity.com/paper/2011.06283/) (2020)
- [FedNS: Improving Federated Learning for collaborative image classification on mobile clients](https://mlsanity.com/paper/2101.07995/) (2021)

## Built on by (most cited first, 19 of 19)

- [FedICT: Federated Multi-task Distillation for Multi-access Edge Computing](https://mlsanity.com/paper/2301.00389/) (2023)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Data](https://mlsanity.com/paper/2405.06413/) (2024)
- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clients](https://mlsanity.com/paper/2304.02892/) (2023)
- [FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data](https://mlsanity.com/paper/2401.08977/) (2024)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data](https://mlsanity.com/paper/2502.19752/) (2025)
- [Dynamic Heterogeneous Federated Learning with Multi-Level Prototypes](https://mlsanity.com/paper/2312.09881/) (2023)
- [You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Data](https://mlsanity.com/paper/2503.06916/) (2025)
- [FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning](https://mlsanity.com/paper/2404.09210/) (2024)
- [FedLF: Adaptive Logit Adjustment and Feature Optimization in Federated Long-Tailed Learning](https://mlsanity.com/paper/2409.12105/) (2024)
- [Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning](https://mlsanity.com/paper/2301.10394/) (2023)
- [Swarm Learning: A Survey of Concepts, Applications, and Trends](https://mlsanity.com/paper/2405.00556/) (2024)
- [Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalization](https://mlsanity.com/paper/2408.07966/) (2024)
- [Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments](https://mlsanity.com/paper/2505.19699/) (2025)
- [Clustered FedStack: Intermediate Global Models with Bayesian Information Criterion](https://mlsanity.com/paper/2309.11044/) (2023)
- [CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Model](https://mlsanity.com/paper/2503.06993/) (2025)
- [FedLTLib: A Comprehensive Benchmark for Federated Long-Tail Learning](https://mlsanity.com/paper/2609.15625/) (2026)

## Abstract (arXiv)

Federated learning provides a privacy guarantee for generating good deep learning models on distributed clients with different kinds of data. Nevertheless, dealing with non-IID data is one of the most challenging problems for federated learning. Researchers have proposed a variety of methods to eliminate the negative influence of non-IIDness. However, they only focus on the non-IID data provided that the universal class distribution is balanced. In many real-world applications, the universal class distribution is long-tailed, which causes the model seriously biased. Therefore, this paper studies the joint problem of non-IID and long-tailed data in federated learning and proposes a corresponding solution called Federated Ensemble Distillation with Imbalance Calibration (FEDIC). To deal with non-IID data, FEDIC uses model ensemble to take advantage of the diversity of models trained on non-IID data. Then, a new distillation method with logit adjustment and calibration gating network is proposed to solve the long-tail problem effectively. We evaluate FEDIC on CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT with a highly non-IID experimental setting, in comparison with the state-of-the-art methods of federated learning and long-tail learning. Our code is available at https://github.com/shangxinyi/FEDIC.