# Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models

arXiv: https://arxiv.org/abs/2008.11364 · HTML: https://mlsanity.com/paper/2008.11364/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 10 of 10)

- [Divergence-aware Federated Self-Supervised Learning](https://mlsanity.com/paper/2204.04385/) (2022)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [Federated Semi-supervised Medical Image Classification via Inter-client Relation Matching](https://mlsanity.com/paper/2106.08600/) (2021)
- [Multimodal Federated Learning on IoT Data](https://mlsanity.com/paper/2109.04833/) (2021)
- [RSCFed: Random Sampling Consensus Federated Semi-supervised Learning](https://mlsanity.com/paper/2203.13993/) (2022)
- [Dynamic Bank Learning for Semi-supervised Federated Image Diagnosis with Class Imbalance](https://mlsanity.com/paper/2206.13079/) (2022)
- [FedCon: A Contrastive Framework for Federated Semi-Supervised Learning](https://mlsanity.com/paper/2109.04533/) (2021)
- [Federated Learning with Positive and Unlabeled Data](https://mlsanity.com/paper/2106.10904/) (2021)
- [Federated Semi-Supervised Learning with Class Distribution Mismatch](https://mlsanity.com/paper/2111.00010/) (2021)
- [Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learning](https://mlsanity.com/paper/2210.00053/) (2022)

## Abstract (arXiv)

Federated learning (FL) is a promising way to use the computing power of mobile devices while maintaining the privacy of users. Current work in FL, however, makes the unrealistic assumption that the users have ground-truth labels on their devices, while also assuming that the server has neither data nor labels. In this work, we consider the more realistic scenario where the users have only unlabeled data, while the server has some labeled data, and where the amount of labeled data is smaller than the amount of unlabeled data. We call this learning problem semi-supervised federated learning (SSFL). For SSFL, we demonstrate that a critical issue that affects the test accuracy is the large gradient diversity of the models from different users. Based on this, we investigate several design choices. First, we find that the so-called consistency regularization loss (CRL), which is widely used in semi-supervised learning, performs reasonably well but has large gradient diversity. Second, we find that Batch Normalization (BN) increases gradient diversity. Replacing BN with the recently-proposed Group Normalization (GN) can reduce gradient diversity and improve test accuracy. Third, we show that CRL combined with GN still has a large gradient diversity when the number of users is large. Based on these results, we propose a novel grouping-based model averaging method to replace the FedAvg averaging method. Overall, our grouping-based averaging, combined with GN and CRL, achieves better test accuracy than not just a contemporary paper on SSFL in the same settings (>10%), but also four supervised FL algorithms.