mlsanityData to 7 Oct 2026

Single-round Self-supervised Distributed Learning using Vision Transformer

arXiv 2301.02064 ↗First posted 5 Jan 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite the recent success of deep learning in the field of medicine, the issue of data scarcity is exacerbated by concerns about privacy and data ownership. Distributed learning approaches, including federated learning, have been investigated to address these issues. However, they are hindered by the need for cumbersome communication overheads and weaknesses in privacy protection. To tackle these challenges, we propose a self-supervised masked sampling distillation method for the vision transformer. This method can be implemented without continuous communication and can enhance privacy by utilizing a vision transformer-specific encryption technique. We conducted extensive experiments on two different tasks, which demonstrated the effectiveness of our method. We achieved superior performance compared to the existing distributed learning strategy as well as the fine-tuning only baseline. Furthermore, since the self-supervised model created using our proposed method can achieve a general semantic understanding of the image, we demonstrate its potential as a task-agnostic self-supervised foundation model for various downstream tasks, thereby expanding its applicability in the medical domain.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

31 of 43 references.

References · 31

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  9. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  10. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  11. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  12. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
19 more
  1. 2018How To Backdoor Federated Learningcited by 745
  2. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  3. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  4. 2021Intriguing Properties of Vision Transformerscited by 446
  5. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 266
  6. 2020Threats to Federated Learning: A Surveycited by 171
  7. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 163
  8. 2021Rethinking BiSeNet For Real-time Semantic Segmentationcited by 124
  9. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  10. 2022Do Gradient Inversion Attacks Make Federated Learning Unsafe?cited by 44
  11. 2022Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imagingcited by 39
  12. 2017Radiomics strategies for risk assessment of tumour failure in head-and-neck cancercited by 31
  13. 2020Privacy Analysis of Deep Learning in the Wild: Membership Inference Attacks against Transfer Learningcited by 19
  14. 2021Membership Inference Attacks on Deep Regression Models for Neuroimagingcited by 19
  15. 2022Federated Self-supervised Learning for Heterogeneous Clientscited by 16
  16. 2022Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutationcited by 10
  17. 2019Convolutional Neural Networks on Randomized Datacited by 6
  18. 2022Federated Self-Supervised Contrastive Learning and Masked Autoencoder for Dermatological Disease Diagnosiscited by 5
  19. 2019Privacy-Preserving Machine Learning Using EtC Imagescited by 3

Built on by

Not cited yet.