# CNN-based real-time 2D-3D deformable registration from a single X-ray projection

arXiv: https://arxiv.org/abs/2212.07692 · HTML: https://mlsanity.com/paper/2212.07692/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 25 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 5 of 5)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [DeepDRR -- A Catalyst for Machine Learning in Fluoroscopy-guided Procedures](https://mlsanity.com/paper/1803.08606/) (2018)
- [Adversarial Noise Layer: Regularize Neural Network By Adding Noise](https://mlsanity.com/paper/1805.08000/) (2018)
- [The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective](https://mlsanity.com/paper/2108.02238/) (2021)
- [Real-Time 2D-3D Deformable Registration with Deep Learning and Application to Lung Radiotherapy Targeting](https://mlsanity.com/paper/1807.08388/) (2018)

## Built on by (most cited first, 1 of 1)

- [Learning Expected Appearances for Intraoperative Registration during Neurosurgery](https://mlsanity.com/paper/2310.01735/) (2023)

## Abstract (arXiv)

Purpose: The purpose of this paper is to present a method for real-time 2D-3D non-rigid registration using a single fluoroscopic image. Such a method can find applications in surgery, interventional radiology and radiotherapy. By estimating a three-dimensional displacement field from a 2D X-ray image, anatomical structures segmented in the preoperative scan can be projected onto the 2D image, thus providing a mixed reality view. Methods: A dataset composed of displacement fields and 2D projections of the anatomy is generated from the preoperative scan. From this dataset, a neural network is trained to recover the unknown 3D displacement field from a single projection image. Results: Our method is validated on lung 4D CT data at different stages of the lung deformation. The training is performed on a 3D CT using random (non domain-specific) diffeomorphic deformations, to which perturbations mimicking the pose uncertainty are added. The model achieves a mean TRE over a series of landmarks ranging from 2.3 to 5.5 mm depending on the amplitude of deformation. Conclusion: In this paper, a CNN-based method for real-time 2D-3D non-rigid registration is presented. This method is able to cope with pose estimation uncertainties, making it applicable to actual clinical scenarios, such as lung surgery, where the C-arm pose is planned before the intervention.