# Decentralized Unsupervised Learning of Visual Representations

arXiv: https://arxiv.org/abs/2111.10763 · HTML: https://mlsanity.com/paper/2111.10763/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 21 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](https://mlsanity.com/paper/1907.09173/) (2019)
- [Federated Visual Classification with Real-World Data Distribution](https://mlsanity.com/paper/2003.08082/) (2020)
- [Federated Unsupervised Representation Learning](https://mlsanity.com/paper/2010.08982/) (2020)
- [InstaHide: Instance-hiding Schemes for Private Distributed Learning](https://mlsanity.com/paper/2010.02772/) (2020)
- [Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems](https://mlsanity.com/paper/1901.06455/) (2019)
- [Fast-Convergent Federated Learning](https://mlsanity.com/paper/2007.13137/) (2020)
- [Real-World Image Datasets for Federated Learning](https://mlsanity.com/paper/1910.11089/) (2019)
- [Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast](https://mlsanity.com/paper/2106.03796/) (2021)

## Built on by (most cited first, 15 of 15)

- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [Distributed Contrastive Learning for Medical Image Segmentation](https://mlsanity.com/paper/2208.03808/) (2022)
- [Synthetic Data Can Also Teach: Synthesizing Effective Data for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2202.06464/) (2022)
- [Contrastive Brain Network Learning via Hierarchical Signed Graph Pooling Model](https://mlsanity.com/paper/2207.07650/) (2022)
- [Self-supervised On-device Federated Learning from Unlabeled Streams](https://mlsanity.com/paper/2212.01006/) (2022)
- [QuMoS: A Framework for Preserving Security of Quantum Machine Learning Model](https://mlsanity.com/paper/2304.11511/) (2023)
- [Federated Self-Supervised Contrastive Learning and Masked Autoencoder for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2208.11278/) (2022)
- [All-in-One Tuning and Structural Pruning for Domain-Specific LLMs](https://mlsanity.com/paper/2412.14426/) (2024)
- [Federated Training of Dual Encoding Models on Small Non-IID Client Datasets](https://mlsanity.com/paper/2210.00092/) (2022)
- [Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devices](https://mlsanity.com/paper/2310.09667/) (2023)
- [A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseases](https://mlsanity.com/paper/2412.06212/) (2024)
- [G-Rank: Unsupervised Continuous Learn-to-Rank for Edge Devices in a P2P Network](https://mlsanity.com/paper/2301.12530/) (2023)
- [Federated Self-Supervised Learning of Monocular Depth Estimators for Autonomous Vehicles](https://mlsanity.com/paper/2310.04837/) (2023)
- [Resource-efficient Layer-wise Federated Self-supervised Learning](https://mlsanity.com/paper/2401.11647/) (2024)

## Abstract (arXiv)

Collaborative learning enables distributed clients to learn a shared model for prediction while keeping the training data local on each client. However, existing collaborative learning methods require fully-labeled data for training, which is inconvenient or sometimes infeasible to obtain due to the high labeling cost and the requirement of expertise. The lack of labels makes collaborative learning impractical in many realistic settings. Self-supervised learning can address this challenge by learning from unlabeled data. Contrastive learning (CL), a self-supervised learning approach, can effectively learn visual representations from unlabeled image data. However, the distributed data collected on clients are usually not independent and identically distributed (non-IID) among clients, and each client may only have few classes of data, which degrades the performance of CL and learned representations. To tackle this problem, we propose a collaborative contrastive learning framework consisting of two approaches: feature fusion and neighborhood matching, by which a unified feature space among clients is learned for better data representations. Feature fusion provides remote features as accurate contrastive information to each client for better local learning. Neighborhood matching further aligns each client's local features to the remote features such that well-clustered features among clients can be learned. Extensive experiments show the effectiveness of the proposed framework. It outperforms other methods by 11% on IID data and matches the performance of centralized learning.