# Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces

arXiv: https://arxiv.org/abs/2209.10526 · HTML: https://mlsanity.com/paper/2209.10526/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [On the Convergence of Local Descent Methods in Federated Learning](https://mlsanity.com/paper/1910.14425/) (2019)
- [Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2111.14213/) (2021)
- [Network Lasso: Clustering and Optimization in Large Graphs](https://mlsanity.com/paper/1507.00280/) (2015)
- [Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity](https://mlsanity.com/paper/2105.00562/) (2021)
- [FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution](https://mlsanity.com/paper/2208.09754/) (2022)
- [Federated Learning with Taskonomy for Non-IID Data](https://mlsanity.com/paper/2103.15947/) (2021)
- [Communication-Efficient Distributed SVD via Local Power Iterations](https://mlsanity.com/paper/2002.08014/) (2020)
- [Vertical Federated Principal Component Analysis and Its Kernel Extension on Feature-wise Distributed Data](https://mlsanity.com/paper/2203.01752/) (2022)
- [Federated Learning From Big Data Over Networks](https://mlsanity.com/paper/2010.14159/) (2020)

## Built on by (most cited first, 30 of 36)

- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [Global and Local Prompts Cooperation via Optimal Transport for Federated Learning](https://mlsanity.com/paper/2403.00041/) (2024)
- [RL Fine-Tuning Heals OOD Forgetting in SFT](https://mlsanity.com/paper/2509.12235/) (2025)
- [Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks](https://mlsanity.com/paper/2209.15595/) (2022)
- [Personalized Federated Continual Learning via Multi-granularity Prompt](https://mlsanity.com/paper/2407.00113/) (2024)
- [Optimizing the Collaboration Structure in Cross-Silo Federated Learning](https://mlsanity.com/paper/2306.06508/) (2023)
- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2405.07925/) (2024)
- [When Do Curricula Work in Federated Learning?](https://mlsanity.com/paper/2212.12712/) (2022)
- [FedClust: Optimizing Federated Learning on Non-IID Data through Weight-Driven Client Clustering](https://mlsanity.com/paper/2403.04144/) (2024)
- [FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering](https://mlsanity.com/paper/2407.07124/) (2024)
- [A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design](https://mlsanity.com/paper/2307.15245/) (2023)
- [Clustered Federated Learning based on Nonconvex Pairwise Fusion](https://mlsanity.com/paper/2211.04218/) (2022)
- [Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologies](https://mlsanity.com/paper/2310.05397/) (2023)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [Geometrically Inspired Kernel Machines for Collaborative Learning Beyond Gradient Descent](https://mlsanity.com/paper/2407.04335/) (2024)
- [A survey on Clustered Federated Learning: Taxonomy, Analysis and Applications](https://mlsanity.com/paper/2501.17512/) (2025)
- [Operator-Theoretic Framework for Gradient-Free Federated Learning](https://mlsanity.com/paper/2512.01025/) (2025)
- [Few-for-Many Personalized Federated Learning](https://mlsanity.com/paper/2603.11992/) (2026)
- [FedRAV: Hierarchically Federated Region-Learning for Traffic Object Classification of Autonomous Vehicles](https://mlsanity.com/paper/2411.13979/) (2024)
- [MMiC: Mitigating Modality Incompleteness in Clustered Federated Learning](https://mlsanity.com/paper/2505.06911/) (2025)
- [Distilling A Universal Expert from Clustered Federated Learning](https://mlsanity.com/paper/2506.20285/) (2025)
- [CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddings](https://mlsanity.com/paper/2506.22427/) (2025)
- [DA-PFL: Dynamic Affinity Aggregation for Personalized Federated Learning](https://mlsanity.com/paper/2403.09284/) (2024)
- [Equitable Federated Learning with Activation Clustering](https://mlsanity.com/paper/2410.19207/) (2024)
- [Stratify: Rethinking Federated Learning for Non-IID Data through Balanced Sampling](https://mlsanity.com/paper/2504.13462/) (2025)
- [Tackling Heterogeneity in Federated Learning via Variance-Reduced Boltzmann Sampling within Homogeneous Social Coalitions](https://mlsanity.com/paper/2506.02897/) (2025)
- [Clustered Federated Learning via Embedding Distributions](https://mlsanity.com/paper/2506.07769/) (2025)
- [EBS-CFL: Efficient and Byzantine-robust Secure Clustered Federated Learning](https://mlsanity.com/paper/2506.13612/) (2025)
- [FedBiCross: Personalized One-Shot Federated Learning on Medical Images](https://mlsanity.com/paper/2601.01901/) (2026)

## Abstract (arXiv)

Clustered federated learning (FL) has been shown to produce promising results by grouping clients into clusters. This is especially effective in scenarios where separate groups of clients have significant differences in the distributions of their local data. Existing clustered FL algorithms are essentially trying to group together clients with similar distributions so that clients in the same cluster can leverage each other's data to better perform federated learning. However, prior clustered FL algorithms attempt to learn these distribution similarities indirectly during training, which can be quite time consuming as many rounds of federated learning may be required until the formation of clusters is stabilized. In this paper, we propose a new approach to federated learning that directly aims to efficiently identify distribution similarities among clients by analyzing the principal angles between the client data subspaces. Each client applies a truncated singular value decomposition (SVD) step on its local data in a single-shot manner to derive a small set of principal vectors, which provides a signature that succinctly captures the main characteristics of the underlying distribution. This small set of principal vectors is provided to the server so that the server can directly identify distribution similarities among the clients to form clusters. This is achieved by comparing the similarities of the principal angles between the client data subspaces spanned by those principal vectors. The approach provides a simple, yet effective clustered FL framework that addresses a broad range of data heterogeneity issues beyond simpler forms of Non-IIDness like label skews. Our clustered FL approach also enables convergence guarantees for non-convex objectives. Our code is available at https://github.com/MMorafah/PACFL.