# Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutation

arXiv: https://arxiv.org/abs/2204.03500 · HTML: https://mlsanity.com/paper/2204.03500/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 30 of 37)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Data Poisoning Attacks Against Federated Learning Systems](https://mlsanity.com/paper/2007.08432/) (2020)
- [Threats to Federated Learning: A Survey](https://mlsanity.com/paper/2003.02133/) (2020)
- [Privacy-preserving Federated Brain Tumour Segmentation](https://mlsanity.com/paper/1910.00962/) (2019)
- [Privacy and Robustness in Federated Learning: Attacks and Defenses](https://mlsanity.com/paper/2012.06337/) (2020)
- [Unleashing the Tiger: Inference Attacks on Split Learning](https://mlsanity.com/paper/2012.02670/) (2020)
- [Learning to Detect Malicious Clients for Robust Federated Learning](https://mlsanity.com/paper/2002.00211/) (2020)
- [Federated Learning: Opportunities and Challenges](https://mlsanity.com/paper/2101.05428/) (2021)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)
- [End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things](https://mlsanity.com/paper/2003.13376/) (2020)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneity](https://mlsanity.com/paper/2012.14453/) (2020)

## Built on by (most cited first, 10 of 10)

- [From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcare](https://mlsanity.com/paper/2409.09727/) (2024)
- [Recent Methodological Advances in Federated Learning for Healthcare](https://mlsanity.com/paper/2310.02874/) (2023)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks](https://mlsanity.com/paper/2501.12824/) (2025)
- [Private Knowledge Sharing in Distributed Learning: A Survey](https://mlsanity.com/paper/2402.06682/) (2024)
- [Heterogeneous Federated Learning with Splited Language Model](https://mlsanity.com/paper/2403.16050/) (2024)
- [Task-agnostic Decision Transformer for Multi-type Agent Control with Federated Split Training](https://mlsanity.com/paper/2405.13445/) (2024)
- [Single-round Self-supervised Distributed Learning using Vision Transformer](https://mlsanity.com/paper/2301.02064/) (2023)
- [Privacy-Preserving Vision Transformer Using Images Encrypted with Restricted Random Permutation Matrices](https://mlsanity.com/paper/2408.08529/) (2024)
- [MedSegNet10: A Publicly Accessible Network Repository for Split Federated Medical Image Segmentation](https://mlsanity.com/paper/2503.20830/) (2025)

## Abstract (arXiv)

The widespread application of artificial intelligence in health research is currently hampered by limitations in data availability. Distributed learning methods such as federated learning (FL) and shared learning (SL) are introduced to solve this problem as well as data management and ownership issues with their different strengths and weaknesses. The recent proposal of federated split task-agnostic (FeSTA) learning tries to reconcile the distinct merits of FL and SL by enabling the multi-task collaboration between participants through Vision Transformer (ViT) architecture, but they suffer from higher communication overhead. To address this, here we present a multi-task distributed learning using ViT with random patch permutation. Instead of using a CNN based head as in FeSTA, p-FeSTA adopts a randomly permuting simple patch embedder, improving the multi-task learning performance without sacrificing privacy. Experimental results confirm that the proposed method significantly enhances the benefit of multi-task collaboration, communication efficiency, and privacy preservation, shedding light on practical multi-task distributed learning in the field of medical imaging.