# Collaborative Unsupervised Visual Representation Learning from Decentralized Data

arXiv: https://arxiv.org/abs/2108.06492 · HTML: https://mlsanity.com/paper/2108.06492/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 14 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [Privacy-preserving Federated Brain Tumour Segmentation](https://mlsanity.com/paper/1910.00962/) (2019)
- [Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation](https://mlsanity.com/paper/1810.04304/) (2018)
- [Federated Unsupervised Representation Learning](https://mlsanity.com/paper/2010.08982/) (2020)
- [Performance Optimization for Federated Person Re-identification via Benchmark Analysis](https://mlsanity.com/paper/2008.11560/) (2020)
- [Real-World Image Datasets for Federated Learning](https://mlsanity.com/paper/1910.11089/) (2019)
- [Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective](https://mlsanity.com/paper/2002.11545/) (2020)
- [EasyFL: A Low-code Federated Learning Platform For Dummies](https://mlsanity.com/paper/2105.07603/) (2021)

## Built on by (most cited first, 30 of 69)

- [When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions](https://mlsanity.com/paper/2306.15546/) (2023)
- [Divergence-aware Federated Self-Supervised Learning](https://mlsanity.com/paper/2204.04385/) (2022)
- [FedAvg with Fine Tuning: Local Updates Lead to Representation Learning](https://mlsanity.com/paper/2205.13692/) (2022)
- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering](https://mlsanity.com/paper/2205.11506/) (2022)
- [FedX: Unsupervised Federated Learning with Cross Knowledge Distillation](https://mlsanity.com/paper/2207.09158/) (2022)
- [Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging](https://mlsanity.com/paper/2205.08576/) (2022)
- [CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning](https://mlsanity.com/paper/2204.03880/) (2022)
- [L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2307.07393/) (2023)
- [Multi-metrics adaptively identifies backdoors in Federated learning](https://mlsanity.com/paper/2303.06601/) (2023)
- [Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases](https://mlsanity.com/paper/2203.07345/) (2022)
- [A Hybrid Self-Supervised Learning Framework for Vertical Federated Learning](https://mlsanity.com/paper/2208.08934/) (2022)
- [Federated Self-supervised Learning for Heterogeneous Clients](https://mlsanity.com/paper/2205.12493/) (2022)
- [Distributed Contrastive Learning for Medical Image Segmentation](https://mlsanity.com/paper/2208.03808/) (2022)
- [Re-thinking Federated Active Learning based on Inter-class Diversity](https://mlsanity.com/paper/2303.12317/) (2023)
- [FedMef: Towards Memory-efficient Federated Dynamic Pruning](https://mlsanity.com/paper/2403.14737/) (2024)
- [Federated Self-supervised Learning for Video Understanding](https://mlsanity.com/paper/2207.01975/) (2022)
- [Federated Learning without Full Labels: A Survey](https://mlsanity.com/paper/2303.14453/) (2023)
- [Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data](https://mlsanity.com/paper/2403.16398/) (2024)
- [Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?](https://mlsanity.com/paper/2210.10947/) (2022)
- [FedWon: Triumphing Multi-domain Federated Learning Without Normalization](https://mlsanity.com/paper/2306.05879/) (2023)
- [Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis](https://mlsanity.com/paper/2205.12144/) (2022)
- [Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators](https://mlsanity.com/paper/2307.05358/) (2023)
- [A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applications](https://mlsanity.com/paper/2403.01387/) (2024)
- [Federated Momentum Contrastive Clustering](https://mlsanity.com/paper/2206.05093/) (2022)
- [Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradients](https://mlsanity.com/paper/2405.16304/) (2024)
- [COALA: A Practical and Vision-Centric Federated Learning Platform](https://mlsanity.com/paper/2407.16560/) (2024)
- [Self-supervised On-device Federated Learning from Unlabeled Streams](https://mlsanity.com/paper/2212.01006/) (2022)
- [Contrastive encoder pre-training-based clustered federated learning for heterogeneous data](https://mlsanity.com/paper/2311.16535/) (2023)
- [FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data](https://mlsanity.com/paper/2405.03949/) (2024)

## Abstract (arXiv)

Unsupervised representation learning has achieved outstanding performances using centralized data available on the Internet. However, the increasing awareness of privacy protection limits sharing of decentralized unlabeled image data that grows explosively in multiple parties (e.g., mobile phones and cameras). As such, a natural problem is how to leverage these data to learn visual representations for downstream tasks while preserving data privacy. To address this problem, we propose a novel federated unsupervised learning framework, FedU. In this framework, each party trains models from unlabeled data independently using contrastive learning with an online network and a target network. Then, a central server aggregates trained models and updates clients' models with the aggregated model. It preserves data privacy as each party only has access to its raw data. Decentralized data among multiple parties are normally non-independent and identically distributed (non-IID), leading to performance degradation. To tackle this challenge, we propose two simple but effective methods: 1) We design the communication protocol to upload only the encoders of online networks for server aggregation and update them with the aggregated encoder; 2) We introduce a new module to dynamically decide how to update predictors based on the divergence caused by non-IID. The predictor is the other component of the online network. Extensive experiments and ablations demonstrate the effectiveness and significance of FedU. It outperforms training with only one party by over 5% and other methods by over 14% in linear and semi-supervised evaluation on non-IID data.