# Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning

arXiv: https://arxiv.org/abs/2102.12920 · HTML: https://mlsanity.com/paper/2102.12920/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 211 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 30 of 75)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Federated Learning in Mobile Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/1909.11875/) (2019)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)

## Built on by (most cited first, 30 of 35)

- [FedBABU: Towards Enhanced Representation for Federated Image Classification](https://mlsanity.com/paper/2106.06042/) (2021)
- [Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directions](https://mlsanity.com/paper/2208.03392/) (2022)
- [Federated Domain Generalization: A Survey](https://mlsanity.com/paper/2306.01334/) (2023)
- [FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks](https://mlsanity.com/paper/2210.14226/) (2022)
- [Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases](https://mlsanity.com/paper/2203.07345/) (2022)
- [Federated Learning and Meta Learning: Approaches, Applications, and Directions](https://mlsanity.com/paper/2210.13111/) (2022)
- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [Robust Training of Federated Models with Extremely Label Deficiency](https://mlsanity.com/paper/2402.14430/) (2024)
- [CADIS: Handling Cluster-skewed Non-IID Data in Federated Learning with Clustered Aggregation and Knowledge DIStilled Regularization](https://mlsanity.com/paper/2302.10413/) (2023)
- [Rethinking Semi-Supervised Federated Learning: How to co-train fully-labeled and fully-unlabeled client imaging data](https://mlsanity.com/paper/2310.18815/) (2023)
- [FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2510.20250/) (2025)
- [Bayesian data fusion with shared priors](https://mlsanity.com/paper/2212.07311/) (2022)
- [Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectives](https://mlsanity.com/paper/2307.02140/) (2023)
- [Gradual Federated Learning with Simulated Annealing](https://mlsanity.com/paper/2110.05178/) (2021)
- [Recommender Engine Driven Client Selection in Federated Brain Tumor Segmentation](https://mlsanity.com/paper/2412.20250/) (2024)
- [A survey on Clustered Federated Learning: Taxonomy, Analysis and Applications](https://mlsanity.com/paper/2501.17512/) (2025)
- [GC-Fed: Gradient Centralized Federated Learning with Partial Client Participation](https://mlsanity.com/paper/2503.13180/) (2025)
- [Federated Learning with Classifier Shift for Class Imbalance](https://mlsanity.com/paper/2304.04972/) (2023)
- [Federated Learning in Healthcare: Model Misconducts, Security, Challenges, Applications, and Future Research Directions -- A Systematic Review](https://mlsanity.com/paper/2405.13832/) (2024)
- [Federated Timeline Synthesis: Scalable and Private Methodology For Model Training and Deployment](https://mlsanity.com/paper/2506.23358/) (2025)
- [Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights](https://mlsanity.com/paper/2509.05142/) (2025)
- [TorchFL: A Performant Library for Bootstrapping Federated Learning Experiments](https://mlsanity.com/paper/2211.00735/) (2022)
- [GTFLAT: Game Theory Based Add-On For Empowering Federated Learning Aggregation Techniques](https://mlsanity.com/paper/2212.04103/) (2022)
- [Enhancing IoT Security Against DDoS Attacks through Federated Learning](https://mlsanity.com/paper/2403.10968/) (2024)
- [Heterogeneous Federated Learning with Convolutional and Spiking Neural Networks](https://mlsanity.com/paper/2406.09680/) (2024)
- [Federated Clustering: An Unsupervised Cluster-Wise Training for Decentralized Data Distributions](https://mlsanity.com/paper/2408.10664/) (2024)
- [FedBrain-Distill: Communication-Efficient Federated Brain Tumor Classification Using Ensemble Knowledge Distillation on Non-IID Data](https://mlsanity.com/paper/2409.05359/) (2024)
- [GAI-Enabled Explainable Personalized Federated Semi-Supervised Learning](https://mlsanity.com/paper/2410.08634/) (2024)
- [FL-CLEANER: byzantine and backdoor defense by CLustering Errors of Activation maps in Non-iid fedErated leaRning](https://mlsanity.com/paper/2501.12123/) (2025)
- [Task-Agnostic Federation over Decentralized Data: Research Landscape and Visions](https://mlsanity.com/paper/2503.03140/) (2025)

## Abstract (arXiv)

Federated learning is a new learning paradigm that decouples data collection and model training via multi-party computation and model aggregation. As a flexible learning setting, federated learning has the potential to integrate with other learning frameworks. We conduct a focused survey of federated learning in conjunction with other learning algorithms. Specifically, we explore various learning algorithms to improve the vanilla federated averaging algorithm and review model fusion methods such as adaptive aggregation, regularization, clustered methods, and Bayesian methods. Following the emerging trends, we also discuss federated learning in the intersection with other learning paradigms, termed federated X learning, where X includes multitask learning, meta-learning, transfer learning, unsupervised learning, and reinforcement learning. In addition to reviewing state-of-the-art studies, this paper also identifies key challenges and applications in this field, while also highlighting promising future directions.