# Exploiting Label Skews in Federated Learning with Model Concatenation

arXiv: https://arxiv.org/abs/2312.06290 · HTML: https://mlsanity.com/paper/2312.06290/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 30 of 33)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [SCAFFOLD: Stochastic Controlled Averaging for Federated Learning](https://mlsanity.com/paper/1910.06378/) (2019)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2111.14213/) (2021)
- [Multi-Center Federated Learning: Clients Clustering for Better Personalization](https://mlsanity.com/paper/2108.08647/) (2021)
- [Federated Learning with Label Distribution Skew via Logits Calibration](https://mlsanity.com/paper/2209.00189/) (2022)
- [Robust Federated Learning: The Case of Affine Distribution Shifts](https://mlsanity.com/paper/2006.08907/) (2020)
- [FedGAN: Federated Generative Adversarial Networks for Distributed Data](https://mlsanity.com/paper/2006.07228/) (2020)
- [Privacy Preserving Vertical Federated Learning for Tree-based Models](https://mlsanity.com/paper/2008.06170/) (2020)
- [Federated Bayesian Optimization via Thompson Sampling](https://mlsanity.com/paper/2010.10154/) (2020)
- [Active Federated Learning](https://mlsanity.com/paper/1909.12641/) (2019)
- [Federated Learning with Only Positive Labels](https://mlsanity.com/paper/2004.10342/) (2020)
- [Practical Federated Gradient Boosting Decision Trees](https://mlsanity.com/paper/1911.04206/) (2019)
- [Flexible Clustered Federated Learning for Client-Level Data Distribution Shift](https://mlsanity.com/paper/2108.09749/) (2021)
- [FedSoft: Soft Clustered Federated Learning with Proximal Local Updating](https://mlsanity.com/paper/2112.06053/) (2021)

## Built on by (most cited first, 14 of 14)

- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [Exploring Vacant Classes in Label-Skewed Federated Learning](https://mlsanity.com/paper/2401.02329/) (2024)
- [SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustments](https://mlsanity.com/paper/2405.04875/) (2024)
- [Optimizing Federated Learning by Entropy-Based Client Selection](https://mlsanity.com/paper/2411.01240/) (2024)
- [Exploiting Label Skewness for Spiking Neural Networks in Federated Learning](https://mlsanity.com/paper/2412.17305/) (2024)
- [Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Tasks](https://mlsanity.com/paper/2402.02225/) (2024)
- [High-Energy Concentration for Federated Learning in Frequency Domain](https://mlsanity.com/paper/2509.12630/) (2025)
- [Distribution-Controlled Client Selection to Improve Federated Learning Strategies](https://mlsanity.com/paper/2509.20877/) (2025)
- [Single-Round Clustered Federated Learning via Data Collaboration Analysis for Non-IID Data](https://mlsanity.com/paper/2601.09304/) (2026)
- [The Key of Parameter Skew in Federated Learning](https://mlsanity.com/paper/2408.11278/) (2024)
- [Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clients](https://mlsanity.com/paper/2603.03007/) (2026)
- [From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learning](https://mlsanity.com/paper/2606.10595/) (2026)
- [WHERE to Generate Matters: Budget-Aware Synthetic Augmentation for Label Skewed Federated Learning](https://mlsanity.com/paper/2607.06616/) (2026)
- [FedCC: Towards Addressing Label Distribution Skews in Distillation-Based Federated Learning](https://mlsanity.com/paper/2608.23031/) (2026)

## Abstract (arXiv)

Federated Learning (FL) has emerged as a promising solution to perform deep learning on different data owners without exchanging raw data. However, non-IID data has been a key challenge in FL, which could significantly degrade the accuracy of the final model. Among different non-IID types, label skews have been challenging and common in image classification and other tasks. Instead of averaging the local models in most previous studies, we propose FedConcat, a simple and effective approach that concatenates these local models as the base of the global model to effectively aggregate the local knowledge. To reduce the size of the global model, we adopt the clustering technique to group the clients by their label distributions and collaboratively train a model inside each cluster. We theoretically analyze the advantage of concatenation over averaging by analyzing the information bottleneck of deep neural networks. Experimental results demonstrate that FedConcat achieves significantly higher accuracy than previous state-of-the-art FL methods in various heterogeneous label skew distribution settings and meanwhile has lower communication costs. Our code is publicly available at https://github.com/sjtudyq/FedConcat.