mlsanityData to 14 Oct 2026

Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?

arXiv 2210.10947 ↗First posted 20 Oct 2022cs.LG cs.AI cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Decentralized learning has been advocated and widely deployed to make efficient use of distributed datasets, with an extensive focus on supervised learning (SL) problems. Unfortunately, the majority of real-world data are unlabeled and can be highly heterogeneous across sources. In this work, we carefully study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL), specifically contrastive visual representation learning. We study the effectiveness of a range of contrastive learning algorithms under decentralized learning settings, on relatively large-scale datasets including ImageNet-100, MS-COCO, and a new real-world robotic warehouse dataset. Our experiments show that the decentralized SSL (Dec-SSL) approach is robust to the heterogeneity of decentralized datasets, and learns useful representation for object classification, detection, and segmentation tasks. This robustness makes it possible to significantly reduce communication and reduce the participation ratio of data sources with only minimal drops in performance. Interestingly, using the same amount of data, the representation learned by Dec-SSL can not only perform on par with that learned by centralized SSL which requires communication and excessive data storage costs, but also sometimes outperform representations extracted from decentralized SL which requires extra knowledge about the data labels. Finally, we provide theoretical insights into understanding why data heterogeneity is less of a concern for Dec-SSL objectives, and introduce feature alignment and clustering techniques to develop a new Dec-SSL algorithm that further improves the performance, in the face of highly non-IID data. Our study presents positive evidence to embrace unlabeled data in decentralized learning, and we hope to provide new insights into whether and why decentralized SSL is effective.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 58 of its 79 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

58 of 79 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
38 more
  1. 2017Mask R-CNNcited by 7,277
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  7. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  11. 2017Decoupled Weight Decay Regularizationcited by 1,831
  12. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  13. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  14. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  15. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  16. 2019Contrastive Multiview Codingcited by 1,301
  17. 2016Colorful Image Colorizationcited by 1,114
  18. 2018Federated Learning with Non-IID Datacited by 1,108
  19. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  20. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  21. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  22. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  23. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  24. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  25. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  26. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  27. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  28. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  29. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  30. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  31. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  32. 2016Learning Representations for Automatic Colorizationcited by 251
  33. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  34. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  35. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  36. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  37. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
  38. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 123

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 3; 2026: 1.

12 papers.

  1. 2023Robot Fleet Learning via Policy Mergingcited by 11
  2. 2024A Mutual Information Perspective on Federated Contrastive Learningcited by 7
  3. 2023Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneitycited by 4
  4. 2023Take Your Pick: Enabling Effective Personalized Federated Learning within Low-dimensional Feature Spacecited by 2
  5. 2024Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformationcited by 2
  6. 2024The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustnesscited by 1
  7. 2025Federated Self-Supervised Learning for One-Shot Cross-Modal and Cross-Imaging Technique Segmentationcited by 1
  8. 2026Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Datacited by 1
  9. 2024Resource-efficient Layer-wise Federated Self-supervised Learningcited by 0
  10. 2024MAPL: Model Agnostic Peer-to-peer Learningcited by 0
  11. 2025Fed-REACT: Federated Representation Learning for Heterogeneous and Evolving Datacited by 0
  12. 2025Osmotic Learning: A Self-Supervised Paradigm for Decentralized Contextual Data Representationcited by 0