# Image-level supervision and self-training for transformer-based cross-modality tumor segmentation

arXiv: https://arxiv.org/abs/2309.09246 · HTML: https://mlsanity.com/paper/2309.09246/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 65 references that are in the map.
- First posted on arXiv 17 Sep 2023.

## Builds on (most cited first, 30 of 37)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Domain Adaptation for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2102.09508/) (2021)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [PseudoSeg: Designing Pseudo Labels for Semantic Segmentation](https://mlsanity.com/paper/2010.09713/) (2020)
- [Understanding Self-Training for Gradual Domain Adaptation](https://mlsanity.com/paper/2002.11361/) (2020)
- [The Importance of Skip Connections in Biomedical Image Segmentation](https://mlsanity.com/paper/1608.04117/) (2016)
- [Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation](https://mlsanity.com/paper/1901.08211/) (2019)
- [Cycle Self-Training for Domain Adaptation](https://mlsanity.com/paper/2103.03571/) (2021)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Distribution Matching Losses Can Hallucinate Features in Medical Image Translation](https://mlsanity.com/paper/1805.08841/) (2018)
- [SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1810.06498/) (2018)
- [CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentation](https://mlsanity.com/paper/2201.02831/) (2022)
- [Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation](https://mlsanity.com/paper/1806.07201/) (2018)
- [Source-Free Domain Adaptation for Image Segmentation](https://mlsanity.com/paper/2108.03152/) (2021)
- [A Data-scalable Transformer for Medical Image Segmentation: Architecture, Model Efficiency, and Benchmark](https://mlsanity.com/paper/2203.00131/) (2022)
- [Adapting Off-the-Shelf Source Segmenter for Target Medical Image Segmentation](https://mlsanity.com/paper/2106.12497/) (2021)
- [Data Efficient Unsupervised Domain Adaptation for Cross-Modality Image Segmentation](https://mlsanity.com/paper/1907.02766/) (2019)
- [Unsupervised Domain Adaptation with Variational Approximation for Cardiac Segmentation](https://mlsanity.com/paper/2106.08752/) (2021)
- [Improving Robustness of Deep Learning Based Knee MRI Segmentation: Mixup and Adversarial Domain Adaptation](https://mlsanity.com/paper/1908.04126/) (2019)
- [COSMOS: Cross-Modality Unsupervised Domain Adaptation for 3D Medical Image Segmentation based on Target-aware Domain Translation and Iterative Self-Training](https://mlsanity.com/paper/2203.16557/) (2022)
- [Unsupervised Domain Adaptation in Semantic Segmentation Based on Pixel Alignment and Self-Training](https://mlsanity.com/paper/2109.14219/) (2021)

## Abstract (arXiv)

Deep neural networks are commonly used for automated medical image segmentation, but models will frequently struggle to generalize well across different imaging modalities. This issue is particularly problematic due to the limited availability of annotated data, making it difficult to deploy these models on a larger scale. To overcome these challenges, we propose a new semi-supervised training strategy called MoDATTS. Our approach is designed for accurate cross-modality 3D tumor segmentation on unpaired bi-modal datasets. An image-to-image translation strategy between imaging modalities is used to produce annotated pseudo-target volumes and improve generalization to the unannotated target modality. We also use powerful vision transformer architectures and introduce an iterative self-training procedure to further close the domain gap between modalities. MoDATTS additionally allows the possibility to extend the training to unannotated target data by exploiting image-level labels with an unsupervised objective that encourages the model to perform 3D diseased-to-healthy translation by disentangling tumors from the background. The proposed model achieves superior performance compared to other methods from participating teams in the CrossMoDA 2022 challenge, as evidenced by its reported top Dice score of 0.87+/-0.04 for the VS segmentation. MoDATTS also yields consistent improvements in Dice scores over baselines on a cross-modality brain tumor segmentation task composed of four different contrasts from the BraTS 2020 challenge dataset, where 95% of a target supervised model performance is reached. We report that 99% and 100% of this maximum performance can be attained if 20% and 50% of the target data is additionally annotated, which further demonstrates that MoDATTS can be leveraged to reduce the annotation burden.