mlsanityData to 14 Oct 2026

HeMIS: Hetero-Modal Image Segmentation

arXiv 1607.05194 ↗First posted 18 Jul 2016cs.CVcited by 103

Not on the map. Why?

Abstract (arXiv)

We introduce a deep learning image segmentation framework that is extremely robust to missing imaging modalities. Instead of attempting to impute or synthesize missing data, the proposed approach learns, for each modality, an embedding of the input image into a single latent vector space for which arithmetic operations (such as taking the mean) are well defined. Points in that space, which are averaged over modalities available at inference time, can then be further processed to yield the desired segmentation. As such, any combinatorial subset of available modalities can be provided as input, without having to learn a combinatorial number of imputation models. Evaluated on two neurological MRI datasets (brain tumors and MS lesions), the approach yields state-of-the-art segmentation results when provided with all modalities; moreover, its performance degrades remarkably gracefully when modalities are removed, significantly more so than alternative mean-filling or other synthesis approaches.

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 9; 2023: 15; 2024: 22; 2025: 15; 2026: 29.

103 papers.

Sort:

By area:

  1. 2021SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retrainingcited by 237
  2. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
  3. 2022mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentationcited by 95
  4. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  5. 2023Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Studycited by 64
  6. 2023M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalitiescited by 56
  7. 2021Modality-aware Mutual Learning for Multi-modal Medical Image Segmentationcited by 47
  8. 2023Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modalitycited by 45
  9. 2021ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalitiescited by 43
  10. 2021Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalitiescited by 42
  11. 2023Unified Multi-Modal Image Synthesis for Missing Modality Imputationcited by 40
  12. 2023MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learningcited by 40
38 more
  1. 2022SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalitiescited by 31
  2. 2022ModDrop++: A Dynamic Filter Network with Intra-subject Co-training for Multiple Sclerosis Lesion Segmentation with Missing Modalitiescited by 22
  3. 2022Multi-Modality Cardiac Image Computing: A Surveycited by 19
  4. 2022Medical Image Segmentation on MRI Images with Missing Modalities: A Reviewcited by 18
  5. 2024Robust Multimodal Learning via Representation Decouplingcited by 17
  6. 2024PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Ratescited by 17
  7. 2021Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentationcited by 16
  8. 2022SFusion: Self-attention based N-to-One Multimodal Fusion Blockcited by 14
  9. 2023Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentationcited by 14
  10. 2021Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalitiescited by 13
  11. 2023Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representationscited by 13
  12. 2025Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncodercited by 13
  13. 2023Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitiscited by 11
  14. 2021Multiple Sclerosis Lesion Analysis in Brain Magnetic Resonance Images: Techniques and Clinical Applicationscited by 10
  15. 2022Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentationcited by 10
  16. 2022X-Metric: An N-Dimensional Information-Theoretic Framework for Groupwise Registration and Deep Combined Computingcited by 10
  17. 2023Dynamic Enhancement Network for Partial Multi-modality Person Re-identificationcited by 10
  18. 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
  19. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  20. 2024Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Expertscited by 9
  21. 2024Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentationcited by 8
  22. 2021HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast Imagescited by 5
  23. 2023Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstructioncited by 5
  24. 2024Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Datacited by 5
  25. 2024Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalitiescited by 5
  26. 2024Progressive Growing of Patch Size: Resource-Efficient Curriculum Learning for Dense Prediction Taskscited by 5
  27. 2024Towards the Causal Complete Cause of Multi-Modal Representation Learningcited by 5
  28. 2025DC-Seg: Disentangled Contrastive Learning for Brain Tumor Segmentation with Missing Modalitiescited by 5
  29. 2023CAFS: Class Adaptive Framework for Semi-Supervised Semantic Segmentationcited by 4
  30. 2024Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalizationcited by 4
  31. 2024MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignmentcited by 4
  32. 2021Improving Tuberculosis (TB) Prediction using Synthetically Generated Computed Tomography (CT) Imagescited by 3
  33. 2023CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translationcited by 3
  34. 2024Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representationcited by 3
  35. 2025Multimodal Masked Autoencoder Pre-training for 3D MRI-Based Brain Tumor Analysis with Missing Modalitiescited by 3
  36. 2025UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensationcited by 3
  37. 2021Cohort Bias Adaptation in Aggregated Datasets for Lesion Segmentationcited by 2
  38. 2023Modality-Agnostic Learning for Medical Image Segmentation Using Multi-modality Self-distillationcited by 2

50 most cited of 103 All 103 (JSON)