# Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering

arXiv: https://arxiv.org/abs/2205.11506 · HTML: https://mlsanity.com/paper/2205.11506/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 90 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 30 of 61)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)

## Built on by (most cited first, 30 of 45)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [On the Importance and Applicability of Pre-Training for Federated Learning](https://mlsanity.com/paper/2206.11488/) (2022)
- [Recent Advances on Federated Learning: A Systematic Survey](https://mlsanity.com/paper/2301.01299/) (2023)
- [Personalized Federated Learning under Mixture of Distributions](https://mlsanity.com/paper/2305.01068/) (2023)
- [L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2307.07393/) (2023)
- [A Survey for Federated Learning Evaluations: Goals and Measures](https://mlsanity.com/paper/2308.11841/) (2023)
- [Re-thinking Federated Active Learning based on Inter-class Diversity](https://mlsanity.com/paper/2303.12317/) (2023)
- [Federated Learning without Full Labels: A Survey](https://mlsanity.com/paper/2303.14453/) (2023)
- [Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data](https://mlsanity.com/paper/2403.16398/) (2024)
- [The Future of Consumer Edge-AI Computing](https://mlsanity.com/paper/2210.10514/) (2022)
- [ProtoFL: Unsupervised Federated Learning via Prototypical Distillation](https://mlsanity.com/paper/2307.12450/) (2023)
- [Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generation](https://mlsanity.com/paper/2407.03086/) (2024)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Analyzing Data-Centric Properties for Graph Contrastive Learning](https://mlsanity.com/paper/2208.02810/) (2022)
- [Towards Unbiased Training in Federated Open-world Semi-supervised Learning](https://mlsanity.com/paper/2305.00771/) (2023)
- [A Mutual Information Perspective on Federated Contrastive Learning](https://mlsanity.com/paper/2405.02081/) (2024)
- [Navigating Distribution Shifts in Medical Image Analysis: A Survey](https://mlsanity.com/paper/2411.05824/) (2024)
- [Federated Continual Novel Class Learning](https://mlsanity.com/paper/2312.13500/) (2023)
- [Federated Source-free Domain Adaptation for Classification: Weighted Cluster Aggregation for Unlabeled Data](https://mlsanity.com/paper/2412.13757/) (2024)
- [FedIL: Federated Incremental Learning from Decentralized Unlabeled Data with Convergence Analysis](https://mlsanity.com/paper/2302.11823/) (2023)
- [FedMAE: Federated Self-Supervised Learning with One-Block Masked Auto-Encoder](https://mlsanity.com/paper/2303.11339/) (2023)
- [FORLA: Federated Object-centric Representation Learning with Slot Attention](https://mlsanity.com/paper/2506.02964/) (2025)
- [Exploring Federated Self-Supervised Learning for General Purpose Audio Understanding](https://mlsanity.com/paper/2402.02889/) (2024)
- [Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learning](https://mlsanity.com/paper/2211.07364/) (2022)
- [Masked Autoencoders are Efficient Continual Federated Learners](https://mlsanity.com/paper/2306.03542/) (2023)
- [UFed-GAN: A Secure Federated Learning Framework with Constrained Computation and Unlabeled Data](https://mlsanity.com/paper/2308.05870/) (2023)
- [Harnessing the Power of Federated Learning in Federated Contextual Bandits](https://mlsanity.com/paper/2312.16341/) (2023)
- [Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributions](https://mlsanity.com/paper/2407.04302/) (2024)
- [Jigsaw Game: Federated Clustering](https://mlsanity.com/paper/2407.12764/) (2024)
- [FedDyMem: Efficient Federated Learning with Dynamic Memory and Memory-Reduce for Unsupervised Image Anomaly Detection](https://mlsanity.com/paper/2502.21012/) (2025)

## Abstract (arXiv)

Federated learning is generally used in tasks where labels are readily available (e.g., next word prediction). Relaxing this constraint requires design of unsupervised learning techniques that can support desirable properties for federated training: robustness to statistical/systems heterogeneity, scalability with number of participants, and communication efficiency. Prior work on this topic has focused on directly extending centralized self-supervised learning techniques, which are not designed to have the properties listed above. To address this situation, we propose Orchestra, a novel unsupervised federated learning technique that exploits the federation's hierarchy to orchestrate a distributed clustering task and enforce a globally consistent partitioning of clients' data into discriminable clusters. We show the algorithmic pipeline in Orchestra guarantees good generalization performance under a linear probe, allowing it to outperform alternative techniques in a broad range of conditions, including variation in heterogeneity, number of clients, participation ratio, and local epochs.