# SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments

arXiv: https://arxiv.org/abs/2206.06127 · HTML: https://mlsanity.com/paper/2206.06127/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 15 of 15)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [DeepDRR -- A Catalyst for Machine Learning in Fluoroscopy-guided Procedures](https://mlsanity.com/paper/1803.08606/) (2018)
- [A Systematic Review of Robustness in Deep Learning for Computer Vision: Mind the gap?](https://mlsanity.com/paper/2112.00639/) (2021)
- [Deep Learning and Conditional Random Fields-based Depth Estimation and Topographical Reconstruction from Conventional Endoscopy](https://mlsanity.com/paper/1710.11216/) (2017)
- [Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration](https://mlsanity.com/paper/1911.07042/) (2019)
- [COVID-19 Infection Map Generation and Detection from Chest X-Ray Images](https://mlsanity.com/paper/2009.12698/) (2020)
- [X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery](https://mlsanity.com/paper/1803.08608/) (2018)
- [The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective](https://mlsanity.com/paper/2108.02238/) (2021)
- [Pose Estimation of Periacetabular Osteotomy Fragments with Intraoperative X-Ray Navigation](https://mlsanity.com/paper/1903.09339/) (2019)
- [A Learning-based Method for Online Adjustment of C-arm Cone-Beam CT Source Trajectories for Artifact Avoidance](https://mlsanity.com/paper/2008.06262/) (2020)
- [Localizing dexterous surgical tools in X-ray for image-based navigation](https://mlsanity.com/paper/1901.06672/) (2019)
- [Multi-task Localization and Segmentation for X-ray Guided Planning in Knee Surgery](https://mlsanity.com/paper/1907.10465/) (2019)
- [Learning to Avoid Poor Images: Towards Task-aware C-arm Cone-beam CT Trajectories](https://mlsanity.com/paper/1909.08868/) (2019)

## Built on by (most cited first, 5 of 5)

- [Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation](https://mlsanity.com/paper/2304.09285/) (2023)
- [Synthetic Data in Radiological Imaging: Current State and Future Outlook](https://mlsanity.com/paper/2407.01561/) (2024)
- [Domain adaptation strategies for 3D reconstruction of the lumbar spine using real fluoroscopy data](https://mlsanity.com/paper/2401.16027/) (2024)
- [Pathological MRI Segmentation by Synthetic Pathological Data Generation in Fetuses and Neonates](https://mlsanity.com/paper/2501.19338/) (2025)
- [EAR: Edge-Aware Reconstruction of 3-D vertebrae structures from bi-planar X-ray images](https://mlsanity.com/paper/2407.20937/) (2024)

## Abstract (arXiv)

Artificial intelligence (AI) now enables automated interpretation of medical images for clinical use. However, AI's potential use for interventional images (versus those involved in triage or diagnosis), such as for guidance during surgery, remains largely untapped. This is because surgical AI systems are currently trained using post hoc analysis of data collected during live surgeries, which has fundamental and practical limitations, including ethical considerations, expense, scalability, data integrity, and a lack of ground truth. Here, we demonstrate that creating realistic simulated images from human models is a viable alternative and complement to large-scale in situ data collection. We show that training AI image analysis models on realistically synthesized data, combined with contemporary domain generalization or adaptation techniques, results in models that on real data perform comparably to models trained on a precisely matched real data training set. Because synthetic generation of training data from human-based models scales easily, we find that our model transfer paradigm for X-ray image analysis, which we refer to as SyntheX, can even outperform real data-trained models due to the effectiveness of training on a larger dataset. We demonstrate the potential of SyntheX on three clinical tasks: Hip image analysis, surgical robotic tool detection, and COVID-19 lung lesion segmentation. SyntheX provides an opportunity to drastically accelerate the conception, design, and evaluation of intelligent systems for X-ray-based medicine. In addition, simulated image environments provide the opportunity to test novel instrumentation, design complementary surgical approaches, and envision novel techniques that improve outcomes, save time, or mitigate human error, freed from the ethical and practical considerations of live human data collection.