mlsanityData to 7 Oct 2026

Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations

arXiv 2309.08747 ↗First posted 15 Sep 2023cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available urlhttps://github.com/ReubenDo/MHVAE.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

24 of 35 references.

References · 24

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2017Squeeze-and-Excitation Networkscited by 4,763
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  7. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 508
  8. 2016Ladder Variational Autoencoderscited by 315
  9. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  10. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  11. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  12. 2019BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modelingcited by 109
12 more
  1. 2016HeMIS: Hetero-Modal Image Segmentationcited by 102
  2. 2019Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Networkcited by 98
  3. 2015Hierarchical Variational Modelscited by 89
  4. 2020Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesiscited by 86
  5. 2019Disentangled Representation Learning in Cardiac Image Analysiscited by 69
  6. 2019Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentationcited by 68
  7. 2022CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentationcited by 58
  8. 2019DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesiscited by 36
  9. 2019CollaGAN : Collaborative GAN for Missing Image Data Imputationcited by 33
  10. 2020Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesiscited by 18
  11. 2020Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasetscited by 15
  12. 2018PIMMS: Permutation Invariant Multi-Modal Segmentationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challengescited by 24
  2. 2024Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Expertscited by 8
  3. 2024Patient-Specific Real-Time Segmentation in Trackerless Brain Ultrasoundcited by 7
  4. 2024Learning to Match 2D Keypoints Across Preoperative MR and Intraoperative Ultrasoundcited by 5
  5. 2025Enhancing Fetal Plane Classification Accuracy with Data Augmentation Using Diffusion Modelscited by 5
  6. 2025A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registrationcited by 3
  7. 2025The Brain Resection Multimodal Image Registration (ReMIND2Reg) 2025 Challengecited by 3
  8. 2026A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgerycited by 2
  9. 2025Influence of High-Performance Image-to-Image Translation Networks on Clinical Visual Assessment and Outcome Prediction: Utilizing Ultrasound to MRI Translation in Prostate Cancercited by 1
  10. 2026From Baseline to Follow-Up: Counterfactual Spine DXA Image Synthesis in UK Biobank Using a Causal Hierarchical Variational Autoencodercited by 0
  11. 2026Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Datacited by 0
  12. 2026What neurosurgeons need to see: synthetic intra-operative MRI from ultrasound for brain-shift compensation in brain tumour surgerycited by 0
1 more
  1. 2026CrossFeat: Bridging Imaging Modalities in Feature Descriptor Spacecited by 0