# The Diversity Bonus: Learning from Dissimilar Distributed Clients in Personalized Federated Learning

arXiv: https://arxiv.org/abs/2407.15464 · HTML: https://mlsanity.com/paper/2407.15464/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 22 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Federated Learning of a Mixture of Global and Local Models](https://mlsanity.com/paper/2002.05516/) (2020)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [Personalized Federated Learning with First Order Model Optimization](https://mlsanity.com/paper/2012.08565/) (2020)
- [Federated Meta-Learning with Fast Convergence and Efficient Communication](https://mlsanity.com/paper/1802.07876/) (2018)
- [Layer-wised Model Aggregation for Personalized Federated Learning](https://mlsanity.com/paper/2205.03993/) (2022)
- [Preservation of the Global Knowledge by Not-True Distillation in Federated Learning](https://mlsanity.com/paper/2106.03097/) (2021)
- [Federated Mutual Learning](https://mlsanity.com/paper/2006.16765/) (2020)
- [Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning](https://mlsanity.com/paper/2110.08394/) (2021)
- [Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing](https://mlsanity.com/paper/2007.09236/) (2020)
- [Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey](https://mlsanity.com/paper/1507.01030/) (2015)
- [Multi-Level Branched Regularization for Federated Learning](https://mlsanity.com/paper/2207.06936/) (2022)
- [Efficient Neural Network Training via Forward and Backward Propagation Sparsification](https://mlsanity.com/paper/2111.05685/) (2021)
- [Personalized Federated Learning With Graph](https://mlsanity.com/paper/2203.00829/) (2022)
- [Personalized Federated Learning with Contextualized Generalization](https://mlsanity.com/paper/2106.13044/) (2021)

## Built on by (most cited first, 3 of 3)

- [DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representations](https://mlsanity.com/paper/2407.17754/) (2024)
- [Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learning](https://mlsanity.com/paper/2503.13543/) (2025)
- [Personalized Digital Health Modeling with Adaptive Support Users](https://mlsanity.com/paper/2605.02004/) (2026)

## Abstract (arXiv)

Personalized Federated Learning (PFL) is a commonly used framework that allows clients to collaboratively train their personalized models. PFL is particularly useful for handling situations where data from different clients are not independent and identically distributed (non-IID). Previous research in PFL implicitly assumes that clients can gain more benefits from those with similar data distributions. Correspondingly, methods such as personalized weight aggregation are developed to assign higher weights to similar clients during training. We pose a question: can a client benefit from other clients with dissimilar data distributions and if so, how? This question is particularly relevant in scenarios with a high degree of non-IID, where clients have widely different data distributions, and learning from only similar clients will lose knowledge from many other clients. We note that when dealing with clients with similar data distributions, methods such as personalized weight aggregation tend to enforce their models to be close in the parameter space. It is reasonable to conjecture that a client can benefit from dissimilar clients if we allow their models to depart from each other. Based on this idea, we propose DiversiFed which allows each client to learn from clients with diversified data distribution in personalized federated learning. DiversiFed pushes personalized models of clients with dissimilar data distributions apart in the parameter space while pulling together those with similar distributions. In addition, to achieve the above effect without using prior knowledge of data distribution, we design a loss function that leverages the model similarity to determine the degree of attraction and repulsion between any two models. Experiments on several datasets show that DiversiFed can benefit from dissimilar clients and thus outperform the state-of-the-art methods.