# Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training

arXiv: https://arxiv.org/abs/2111.01338 · HTML: https://mlsanity.com/paper/2111.01338/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 71 references that are in the map.
- First posted on arXiv 2 Nov 2021.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning](https://mlsanity.com/paper/1812.00535/) (2018)
- [A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises](https://mlsanity.com/paper/2008.09104/) (2020)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)

## Built on by (most cited first, 17 of 17)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [FedTP: Federated Learning by Transformer Personalization](https://mlsanity.com/paper/2211.01572/) (2022)
- [Predictive GAN-powered Multi-Objective Optimization for Hybrid Federated Split Learning](https://mlsanity.com/paper/2209.02428/) (2022)
- [Recent Methodological Advances in Federated Learning for Healthcare](https://mlsanity.com/paper/2310.02874/) (2023)
- [SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimization](https://mlsanity.com/paper/2410.10759/) (2024)
- [Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutation](https://mlsanity.com/paper/2204.03500/) (2022)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [Vision Transformers in Medical Imaging: A Review](https://mlsanity.com/paper/2211.10043/) (2022)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [FeSViBS: Federated Split Learning of Vision Transformer with Block Sampling](https://mlsanity.com/paper/2306.14638/) (2023)
- [Multi-Modal Federated Learning for Cancer Staging over Non-IID Datasets with Unbalanced Modalities](https://mlsanity.com/paper/2401.03609/) (2024)
- [Privacy-preserving machine learning for healthcare: open challenges and future perspectives](https://mlsanity.com/paper/2303.15563/) (2023)
- [SLPerf: a Unified Framework for Benchmarking Split Learning](https://mlsanity.com/paper/2304.01502/) (2023)
- [Federated Split Learning with Improved Communication and Storage Efficiency](https://mlsanity.com/paper/2507.15816/) (2025)
- [Heterogeneous Federated Learning with Splited Language Model](https://mlsanity.com/paper/2403.16050/) (2024)
- [MedSegNet10: A Publicly Accessible Network Repository for Split Federated Medical Image Segmentation](https://mlsanity.com/paper/2503.20830/) (2025)
- [MTMed3D: A Multi-Task Transformer-Based Model for 3D Medical Imaging](https://mlsanity.com/paper/2511.12373/) (2025)

## Abstract (arXiv)

Federated learning, which shares the weights of the neural network across clients, is gaining attention in the healthcare sector as it enables training on a large corpus of decentralized data while maintaining data privacy. For example, this enables neural network training for COVID-19 diagnosis on chest X-ray (CXR) images without collecting patient CXR data across multiple hospitals. Unfortunately, the exchange of the weights quickly consumes the network bandwidth if highly expressive network architecture is employed. So-called split learning partially solves this problem by dividing a neural network into a client and a server part, so that the client part of the network takes up less extensive computation resources and bandwidth. However, it is not clear how to find the optimal split without sacrificing the overall network performance. To amalgamate these methods and thereby maximize their distinct strengths, here we show that the Vision Transformer, a recently developed deep learning architecture with straightforward decomposable configuration, is ideally suitable for split learning without sacrificing performance. Even under the non-independent and identically distributed data distribution which emulates a real collaboration between hospitals using CXR datasets from multiple sources, the proposed framework was able to attain performance comparable to data-centralized training. In addition, the proposed framework along with heterogeneous multi-task clients also improves individual task performances including the diagnosis of COVID-19, eliminating the need for sharing large weights with innumerable parameters. Our results affirm the suitability of Transformer for collaborative learning in medical imaging and pave the way forward for future real-world implementations.