# What neurosurgeons need to see: synthetic intra-operative MRI from ultrasound for brain-shift compensation in brain tumour surgery

arXiv: https://arxiv.org/abs/2606.07658 · HTML: https://mlsanity.com/paper/2606.07658/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 18 of 18)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [SynthMorph: learning contrast-invariant registration without acquired images](https://mlsanity.com/paper/2004.10282/) (2020)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)
- [Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis](https://mlsanity.com/paper/2008.08698/) (2020)
- [Evaluation of MRI to ultrasound registration methods for brain shift correction: The CuRIOUS2018 Challenge](https://mlsanity.com/paper/1904.10535/) (2019)
- [Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations](https://mlsanity.com/paper/2309.08747/) (2023)
- [Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts](https://mlsanity.com/paper/2410.19378/) (2024)
- [Learn2Reg 2024: New Benchmark Datasets Driving Progress on New Challenges](https://mlsanity.com/paper/2509.01217/) (2025)
- [Unsupervised Multimodal 3D Medical Image Registration with Multilevel Correlation Balanced Optimization](https://mlsanity.com/paper/2409.05040/) (2024)
- [Translation of Fetal Brain Ultrasound Images into Pseudo-MRI Images using Artificial Intelligence](https://mlsanity.com/paper/2504.02408/) (2025)
- [A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration](https://mlsanity.com/paper/2507.18551/) (2025)
- [Unsupervised MR-US Multimodal Image Registration with Multilevel Correlation Pyramidal Optimization](https://mlsanity.com/paper/2602.06288/) (2026)
- [BrainVoxGen: Deep learning framework for synthesis of Ultrasound to MRI](https://mlsanity.com/paper/2310.08608/) (2023)
- [DiffUS: Differentiable Ultrasound Rendering from Volumetric Imaging](https://mlsanity.com/paper/2508.06768/) (2025)
- [A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgery](https://mlsanity.com/paper/2606.00630/) (2026)

## Abstract (arXiv)

Maximal safe resection is the primary objective in glioma surgery. Neuronavigation guidance is progressively degraded by brain shift after dural opening. Intraoperative MRI can compensate but needs dedicated infrastructure and is rarely available, whereas intraoperative ultrasound (ioUS) is inexpensive, repeatable, and compatible with routine workflows. Navigation systems combining ioUS with preoperative MRI usually rely on rigid registration; even deformable multimodal registration is limited by ultrasound speckle contrast, a narrow field of view, and the inability to represent structures absent from the preoperative scan, most critically the resection cavity and residual tumor. We propose an end-to-end pipeline that generates a new whole-brain MRI volume in the preoperative imaging space by merging the preoperative MRI, a synthetic MRI generated from the ioUS, and a deformable registration anchored on that synthetic image. It integrates a 2.5D residual-transformer synthesis backbone (ResViT-2.5D) and a two-stage registration coupling NiftyReg with a synthesis-anchored SynthMorph stage, operating directly on raw scanner inputs. On a post-resection ReMIND cohort, ResViT-2.5D produced synthetic images closely matching the intraoperative T2 across structural, intensity, and perceptual metrics. In 14 subjects with 215 expert landmarks, the synthesis-anchored registration reduced the mean target registration error from 6.27 to 5.86 mm, matching a strong classical NiftyReg baseline (5.85 mm) while yielding a diffeomorphic deformation field in every subject. The contribution is not a gain in registration accuracy but the integrated volume itself, which inside the ultrasound field of view it reflects the intraoperative post-resection state. This provides the surgeon with an MRI-like update of the operative field with potential for integration into surgical-navigation workflows.