mlsanityData to 7 Oct 2026

Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts

arXiv 2410.19378 ↗First posted 25 Oct 2024cs.CV cs.LG eess.IVcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

33 of 56 references.

References · 33

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2017Squeeze-and-Excitation Networkscited by 4,763
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  7. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  8. 2016Ladder Variational Autoencoderscited by 315
  9. 2021SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retrainingcited by 235
  10. 2022BBDM: Image-to-image Translation with Brownian Bridge Diffusion Modelscited by 222
  11. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  12. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
21 more
  1. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  2. 2023On Memorization in Diffusion Modelscited by 122
  3. 2019BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modelingcited by 109
  4. 2016HeMIS: Hetero-Modal Image Segmentationcited by 102
  5. 2019Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Networkcited by 98
  6. 2015Hierarchical Variational Modelscited by 89
  7. 2020Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesiscited by 86
  8. 2021Generalized Multimodal ELBOcited by 86
  9. 2019Disentangled Representation Learning in Cardiac Image Analysiscited by 69
  10. 2019Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentationcited by 68
  11. 2023CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesiscited by 64
  12. 2022CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentationcited by 58
  13. 2019DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesiscited by 36
  14. 2019CollaGAN : Collaborative GAN for Missing Image Data Imputationcited by 33
  15. 2022A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completioncited by 23
  16. 2020Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesiscited by 18
  17. 2020Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasetscited by 15
  18. 2023Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representationscited by 13
  19. 2018PIMMS: Permutation Invariant Multi-Modal Segmentationcited by 9
  20. 2024Patient-Specific Real-Time Segmentation in Trackerless Brain Ultrasoundcited by 7
  21. 2022RESECT-SEG: Open access annotations of intra-operative brain tumor ultrasound imagescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Virtual Full-stack Scanning of Brain MRI via Imputing Any Quantised Codecited by 4
  2. 2025A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registrationcited by 3
  3. 2025The Brain Resection Multimodal Image Registration (ReMIND2Reg) 2025 Challengecited by 3
  4. 2026A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgerycited by 2
  5. 2026Data-driven registration and modeling of brain deformation for image-guided neurosurgerycited by 0
  6. 2026MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysiscited by 0
  7. 2026What neurosurgeons need to see: synthetic intra-operative MRI from ultrasound for brain-shift compensation in brain tumour surgerycited by 0
  8. 2026Posterior Samplings are Missing Modalities Generators for Medical Image Translationcited by 0