mlsanityData to 7 Oct 2026

Real World Federated Learning with a Knowledge Distilled Transformer for Cardiac CT Imaging

arXiv 2407.07557 ↗First posted 10 Jul 2024eess.IV cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Federated learning is a renowned technique for utilizing decentralized data while preserving privacy. However, real-world applications often face challenges like partially labeled datasets, where only a few locations have certain expert annotations, leaving large portions of unlabeled data unused. Leveraging these could enhance transformer architectures ability in regimes with small and diversely annotated sets. We conduct the largest federated cardiac CT analysis to date (n=8,104) in a real-world setting across eight hospitals. Our two-step semi-supervised strategy distills knowledge from task-specific CNNs into a transformer. First, CNNs predict on unlabeled data per label type and then the transformer learns from these predictions with label-specific heads. This improves predictive accuracy and enables simultaneous learning of all partial labels across the federation, and outperforms UNet-based models in generalizability on downstream tasks. Code and model weights are made openly available for leveraging future cardiac CT analysis.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 44 of its 89 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

44 of 89 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2020Knowledge Distillation: A Surveycited by 1,578
  11. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  12. 2019Deep Leakage from Gradientscited by 1,004
32 more
  1. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
  2. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
  3. 2020The Future of Digital Health with Federated Learningcited by 625
  4. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 614
  5. 2019Similarity-Preserving Knowledge Distillationcited by 507
  6. 2022MONAI: An open-source framework for deep learning in healthcarecited by 438
  7. 2018Scalable Private Learning with PATEcited by 266
  8. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  9. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  10. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  11. 2020TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learningcited by 176
  12. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  13. 2019Prior-aware Neural Network for Partially-Supervised Multi-Organ Segmentationcited by 88
  14. 2022Federated Learning Enables Big Data for Rare Cancer Boundary Detectioncited by 83
  15. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  16. 2018Fast Human Pose Estimationcited by 68
  17. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  18. 2022ImageCAS: A Large-Scale Dataset and Benchmark for Coronary Artery Segmentation based on Computed Tomography Angiography Imagescited by 54
  19. 2020Federated Knowledge Distillationcited by 51
  20. 2021The Federated Tumor Segmentation (FeTS) Challengecited by 49
  21. 2018Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attackscited by 48
  22. 2023MultiTalent: A Multi-Dataset Approach to Medical Image Segmentationcited by 45
  23. 2018Learning Student Networks via Feature Embeddingcited by 43
  24. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  25. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  26. 2019Segmentations-Leak: Membership Inference Attacks and Defenses in Semantic Image Segmentationcited by 29
  27. 2022Labeling instructions matter in biomedical image analysiscited by 20
  28. 2021Federated Learning for Multi-Center Imaging Diagnostics: A Study in Cardiovascular Diseasecited by 19
  29. 2022Federated Multi-organ Segmentation with Inconsistent Labelscited by 15
  30. 2023ConDistFL: Conditional Distillation for Federated Learning from Partially Annotated Datacited by 6
  31. 2022Content-Aware Differential Privacy with Conditional Invertible Neural Networkscited by 3
  32. 2023Multi-site Organ Segmentation with Federated Partial Supervision and Site Adaptationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 3.

8 papers.

  1. 2025Federated EndoViT: Pretraining Vision Transformers via Federated Learning on Endoscopic Image Collectionscited by 3
  2. 2025Federated Learning for Large Models in Medical Imaging: A Comprehensive Reviewcited by 2
  3. 2025Cardiac-CLIP: A Vision-Language Foundation Model for 3D Cardiac CT Imagescited by 1
  4. 2024Multi-Modal Dataset Creation for Federated Learning with DICOM Structured Reportscited by 0
  5. 2025FRIEREN: Federated Learning with Vision-Language Regularization for Segmentationcited by 0
  6. 2026Overcoming data scarcity through multi-center federated learning for organs-at-risk segmentation in pediatric upper abdominal radiotherapycited by 0
  7. 2026FedMust: Semi-supervised Multi-task Student-Teacher Federated Learning for Multi-organ CT Segmentationcited by 0
  8. 2026A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensionscited by 0