# Unpaired Multi-modal Segmentation via Knowledge Distillation

arXiv: https://arxiv.org/abs/2001.03111 · HTML: https://mlsanity.com/paper/2001.03111/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2020.

## Built on by (most cited first, 30 of 81)

- [Source-Free Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16372/) (2021)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Bidirectional Copy-Paste for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2305.00673/) (2023)
- [A Comprehensive Survey on Source-free Domain Adaptation](https://mlsanity.com/paper/2302.11803/) (2023)
- [Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2304.10410/) (2023)
- [UniMiSS: Universal Medical Self-Supervised Learning via Breaking Dimensionality Barrier](https://mlsanity.com/paper/2112.09356/) (2021)
- [EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation](https://mlsanity.com/paper/2111.12341/) (2021)
- [Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2310.01035/) (2023)
- [Efficient Medical Image Segmentation Based on Knowledge Distillation](https://mlsanity.com/paper/2108.09987/) (2021)
- [Federated Semi-supervised Medical Image Classification via Inter-client Relation Matching](https://mlsanity.com/paper/2106.08600/) (2021)
- [Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation](https://mlsanity.com/paper/2205.00858/) (2022)
- [A Closer Look at Multimodal Representation Collapse](https://mlsanity.com/paper/2505.22483/) (2025)
- [Incremental Learning Meets Transfer Learning: Application to Multi-site Prostate MRI Segmentation](https://mlsanity.com/paper/2206.01369/) (2022)
- [Multi-Modality Cardiac Image Computing: A Survey](https://mlsanity.com/paper/2208.12881/) (2022)
- [Weakly supervised segmentation with cross-modality equivariant constraints](https://mlsanity.com/paper/2104.02488/) (2021)
- [Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation](https://mlsanity.com/paper/2407.03056/) (2024)
- [PA-Seg: Learning from Point Annotations for 3D Medical Image Segmentation using Contextual Regularization and Cross Knowledge Distillation](https://mlsanity.com/paper/2208.05669/) (2022)
- [Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction](https://mlsanity.com/paper/2207.10851/) (2022)
- [Deep Multimodal Guidance for Medical Image Classification](https://mlsanity.com/paper/2203.05683/) (2022)
- [Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency](https://mlsanity.com/paper/2206.10571/) (2022)
- [MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data](https://mlsanity.com/paper/2104.03618/) (2021)
- [Dual-Teacher++: Exploiting Intra-domain and Inter-domain Knowledge with Reliable Transfer for Cardiac Segmentation](https://mlsanity.com/paper/2101.02375/) (2021)
- [Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data](https://mlsanity.com/paper/2510.22880/) (2025)
- [Learning by Hallucinating: Vision-Language Pre-training with Weak Supervision](https://mlsanity.com/paper/2210.13591/) (2022)
- [Uncertainty-aware Evidential Fusion-based Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2404.06177/) (2024)
- [MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training](https://mlsanity.com/paper/2411.15576/) (2024)
- [Deep Class-Specific Affinity-Guided Convolutional Network for Multimodal Unpaired Image Segmentation](https://mlsanity.com/paper/2101.01513/) (2021)
- [Multi-Task, Multi-Domain Deep Segmentation with Shared Representations and Contrastive Regularization for Sparse Pediatric Datasets](https://mlsanity.com/paper/2105.10310/) (2021)
- [Fighting the scanner effect in brain MRI segmentation with a progressive level-of-detail network trained on multi-site data](https://mlsanity.com/paper/2211.02400/) (2022)
- [Medical Image Segmentation with Domain Adaptation: A Survey](https://mlsanity.com/paper/2311.01702/) (2023)

## Abstract (arXiv)

Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model, we introduce a novel loss term inspired by knowledge distillation, by explicitly constraining the KL-divergence of our derived prediction distributions between modalities. We have extensively validated our approach on two multi-class segmentation problems: i) cardiac structure segmentation, and ii) abdominal organ segmentation. Different network settings, i.e., 2D dilated network and 3D U-net, are utilized to investigate our method's general efficacy. Experimental results on both tasks demonstrate that our novel multi-modal learning scheme consistently outperforms single-modal training and previous multi-modal approaches.