# Translating Simulation Images to X-ray Images via Multi-Scale Semantic Matching

arXiv: https://arxiv.org/abs/2304.07693 · HTML: https://mlsanity.com/paper/2304.07693/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 16 Apr 2023.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation](https://mlsanity.com/paper/1908.10454/) (2019)
- [GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification](https://mlsanity.com/paper/1803.01229/) (2018)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [MedGAN: Medical Image Translation using GANs](https://mlsanity.com/paper/1806.06397/) (2018)
- [Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1809.05852/) (2018)
- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](https://mlsanity.com/paper/1612.05362/) (2016)
- [One-Sided Unsupervised Domain Mapping](https://mlsanity.com/paper/1706.00826/) (2017)
- [Deep MR to CT Synthesis using Unpaired Data](https://mlsanity.com/paper/1708.01155/) (2017)
- [Breaking the Dilemma of Medical Image-to-image Translation](https://mlsanity.com/paper/2110.06465/) (2021)
- [The Spatially-Correlative Loss for Various Image Translation Tasks](https://mlsanity.com/paper/2104.00854/) (2021)
- [Learning Implicit Brain MRI Manifolds with Deep Learning](https://mlsanity.com/paper/1801.01847/) (2018)
- [End-to-End Real-time Catheter Segmentation with Optical Flow-Guided Warping during Endovascular Intervention](https://mlsanity.com/paper/2006.09117/) (2020)
- [Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation](https://mlsanity.com/paper/2107.06941/) (2021)
- [Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge](https://mlsanity.com/paper/2110.01293/) (2021)
- [Bridging the gap between paired and unpaired medical image translation](https://mlsanity.com/paper/2110.08407/) (2021)
- [Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging](https://mlsanity.com/paper/1911.08163/) (2019)
- [Flow-based Deformation Guidance for Unpaired Multi-Contrast MRI Image-to-Image Translation](https://mlsanity.com/paper/2012.01777/) (2020)
- [Goldilocks-curriculum Domain Randomization and Fractal Perlin Noise with Application to Sim2Real Pneumonia Lesion Detection](https://mlsanity.com/paper/2204.13849/) (2022)
- [sim2real: Cardiac MR Image Simulation-to-Real Translation via Unsupervised GANs](https://mlsanity.com/paper/2208.04874/) (2022)

## Built on by (most cited first, 1 of 1)

- [Autonomous Catheterization with Open-source Simulator and Expert Trajectory](https://mlsanity.com/paper/2401.09059/) (2024)

## Abstract (arXiv)

Endovascular intervention training is increasingly being conducted in virtual simulators. However, transferring the experience from endovascular simulators to the real world remains an open problem. The key challenge is the virtual environments are usually not realistically simulated, especially the simulation images. In this paper, we propose a new method to translate simulation images from an endovascular simulator to X-ray images. Previous image-to-image translation methods often focus on visual effects and neglect structure information, which is critical for medical images. To address this gap, we propose a new method that utilizes multi-scale semantic matching. We apply self-domain semantic matching to ensure that the input image and the generated image have the same positional semantic relationships. We further apply cross-domain matching to eliminate the effects of different styles. The intensive experiment shows that our method generates realistic X-ray images and outperforms other state-of-the-art approaches by a large margin. We also collect a new large-scale dataset to serve as the new benchmark for this task. Our source code and dataset will be made publicly available.