# FedX: Unsupervised Federated Learning with Cross Knowledge Distillation

arXiv: https://arxiv.org/abs/2207.09158 · HTML: https://mlsanity.com/paper/2207.09158/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 36 of its 47 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Representation Learning via Invariant Causal Mechanisms](https://mlsanity.com/paper/2010.07922/) (2020)
- [Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation](https://mlsanity.com/paper/1810.04304/) (2018)
- [FedVision: An Online Visual Object Detection Platform Powered by Federated Learning](https://mlsanity.com/paper/2001.06202/) (2020)
- [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020)
- [Federated Unsupervised Representation Learning](https://mlsanity.com/paper/2010.08982/) (2020)
- [Collaborative Unsupervised Visual Representation Learning from Decentralized Data](https://mlsanity.com/paper/2108.06492/) (2021)
- [Improving Unsupervised Image Clustering With Robust Learning](https://mlsanity.com/paper/2012.11150/) (2020)

## Built on by (most cited first, 30 of 40)

- [Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges](https://mlsanity.com/paper/2211.08413/) (2022)
- [Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.03457/) (2023)
- [FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models](https://mlsanity.com/paper/2406.04845/) (2024)
- [A Hybrid Self-Supervised Learning Framework for Vertical Federated Learning](https://mlsanity.com/paper/2208.08934/) (2022)
- [Federated Distillation: A Survey](https://mlsanity.com/paper/2404.08564/) (2024)
- [Federated Learning without Full Labels: A Survey](https://mlsanity.com/paper/2303.14453/) (2023)
- [Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data](https://mlsanity.com/paper/2403.16398/) (2024)
- [FedDefender: Client-Side Attack-Tolerant Federated Learning](https://mlsanity.com/paper/2307.09048/) (2023)
- [ProtoFL: Unsupervised Federated Learning via Prototypical Distillation](https://mlsanity.com/paper/2307.12450/) (2023)
- [FeDXL: Provable Federated Learning for Deep X-Risk Optimization](https://mlsanity.com/paper/2210.14396/) (2022)
- [Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions](https://mlsanity.com/paper/2406.10861/) (2024)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Fast-Convergent and Communication-Alleviated Heterogeneous Hierarchical Federated Learning in Autonomous Driving](https://mlsanity.com/paper/2409.19560/) (2024)
- [Towards Attack-tolerant Federated Learning via Critical Parameter Analysis](https://mlsanity.com/paper/2308.09318/) (2023)
- [Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradients](https://mlsanity.com/paper/2405.16304/) (2024)
- [Federated Retrieval-Augmented Generation: A Systematic Mapping Study](https://mlsanity.com/paper/2505.18906/) (2025)
- [FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data](https://mlsanity.com/paper/2405.03949/) (2024)
- [Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization](https://mlsanity.com/paper/2309.06692/) (2023)
- [CCFC: Bridging Federated Clustering and Contrastive Learning](https://mlsanity.com/paper/2401.06634/) (2024)
- [FORLA: Federated Object-centric Representation Learning with Slot Attention](https://mlsanity.com/paper/2506.02964/) (2025)
- [FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learning](https://mlsanity.com/paper/2507.06482/) (2025)
- [Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practice](https://mlsanity.com/paper/2510.12595/) (2025)
- [Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learning](https://mlsanity.com/paper/2211.07364/) (2022)
- [Masked Autoencoders are Efficient Continual Federated Learners](https://mlsanity.com/paper/2306.03542/) (2023)
- [UFed-GAN: A Secure Federated Learning Framework with Constrained Computation and Unlabeled Data](https://mlsanity.com/paper/2308.05870/) (2023)
- [Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity](https://mlsanity.com/paper/2404.03854/) (2024)
- [FedDyMem: Efficient Federated Learning with Dynamic Memory and Memory-Reduce for Unsupervised Image Anomaly Detection](https://mlsanity.com/paper/2502.21012/) (2025)
- [Hierarchical Knowledge Structuring for Effective Federated Learning in Heterogeneous Environments](https://mlsanity.com/paper/2504.03505/) (2025)
- [Federated Unsupervised Visual Representation Learning via Exploiting General Content and Personal Style](https://mlsanity.com/paper/2211.06470/) (2022)
- [Federated Semi-Supervised Learning with Annotation Heterogeneity](https://mlsanity.com/paper/2303.02445/) (2023)

## Abstract (arXiv)

This paper presents FedX, an unsupervised federated learning framework. Our model learns unbiased representation from decentralized and heterogeneous local data. It employs a two-sided knowledge distillation with contrastive learning as a core component, allowing the federated system to function without requiring clients to share any data features. Furthermore, its adaptable architecture can be used as an add-on module for existing unsupervised algorithms in federated settings. Experiments show that our model improves performance significantly (1.58--5.52pp) on five unsupervised algorithms.