# Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering

arXiv: https://arxiv.org/abs/2312.06358 · HTML: https://mlsanity.com/paper/2312.06358/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 60 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Understanding the Limitations of CNN-based Absolute Camera Pose Regression](https://mlsanity.com/paper/1903.07504/) (2019)
- [An Unsupervised Learning Model for Deformable Medical Image Registration](https://mlsanity.com/paper/1802.02604/) (2018)
- [Learning Less is More - 6D Camera Localization via 3D Surface Regression](https://mlsanity.com/paper/1711.10228/) (2017)
- [Image-based localization using LSTMs for structured feature correlation](https://mlsanity.com/paper/1611.07890/) (2016)
- [DeepDRR -- A Catalyst for Machine Learning in Fluoroscopy-guided Procedures](https://mlsanity.com/paper/1803.08606/) (2018)
- [Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging](https://mlsanity.com/paper/2208.12737/) (2022)
- [A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty](https://mlsanity.com/paper/2006.01031/) (2020)
- [Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration](https://mlsanity.com/paper/1911.07042/) (2019)
- [The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective](https://mlsanity.com/paper/2108.02238/) (2021)
- [Self-Supervised 2D/3D Registration for X-Ray to CT Image Fusion](https://mlsanity.com/paper/2210.07611/) (2022)
- [Pose Estimation of Periacetabular Osteotomy Fragments with Intraoperative X-Ray Navigation](https://mlsanity.com/paper/1903.09339/) (2019)
- [Patch-Based Image Similarity for Intraoperative 2D/3D Pelvis Registration During Periacetabular Osteotomy](https://mlsanity.com/paper/1909.10443/) (2019)
- [X-Ray to CT Rigid Registration Using Scene Coordinate Regression](https://mlsanity.com/paper/2311.15087/) (2023)
- [Deep Iterative 2D/3D Registration](https://mlsanity.com/paper/2107.10004/) (2021)

## Built on by (most cited first, 24 of 24)

- [DDGS-CT: Direction-Disentangled Gaussian Splatting for Realistic Volume Rendering](https://mlsanity.com/paper/2406.02518/) (2024)
- [RayEmb: Arbitrary Landmark Detection in X-Ray Images Using Ray Embedding Subspace](https://mlsanity.com/paper/2410.08152/) (2024)
- [An Optimization-based Baseline for Rigid 2D/3D Registration Applied to Spine Surgical Navigation Using CMA-ES](https://mlsanity.com/paper/2402.05642/) (2024)
- [Differentiable Voxel-based X-ray Rendering Improves Sparse-View 3D CBCT Reconstruction](https://mlsanity.com/paper/2411.19224/) (2024)
- [PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations](https://mlsanity.com/paper/2505.19256/) (2025)
- [DiffUS: Differentiable Ultrasound Rendering from Volumetric Imaging](https://mlsanity.com/paper/2508.06768/) (2025)
- [Automatic Multi-View X-Ray/CT Registration Using Bone Substructure Contours](https://mlsanity.com/paper/2506.13292/) (2025)
- [Intraoperative 2D/3D Registration via Spherical Similarity Learning and Differentiable Levenberg-Marquardt Optimization](https://mlsanity.com/paper/2509.06890/) (2025)
- [CardioComposer: Leveraging Differentiable Geometry for Compositional Control of Anatomical Diffusion Models](https://mlsanity.com/paper/2509.08015/) (2025)
- [LayersReg: A Layer-by-Layer Progressive Regressor for Reliable Intraoperative 3D/2D Registration](https://mlsanity.com/paper/2606.26647/) (2026)
- [Rapid patient-specific neural networks for intraoperative X-ray to volume registration](https://mlsanity.com/paper/2503.16309/) (2025)
- [Robust and Accurate Multi-view 2D/3D Image Registration with Differentiable X-ray Rendering and Dual Cross-view Constraints](https://mlsanity.com/paper/2506.22191/) (2025)
- [RadGS-Reg: Registering Spine CT with Biplanar X-rays via Joint 3D Radiative Gaussians Reconstruction and 3D/3D Registration](https://mlsanity.com/paper/2508.21154/) (2025)
- [Veriserum: A dual-plane fluoroscopic dataset with knee implant phantoms for deep learning in medical imaging](https://mlsanity.com/paper/2509.05483/) (2025)
- [Silhouette-to-Contour Registration: Aligning Intraoral Scan Models with Cephalometric Radiographs](https://mlsanity.com/paper/2511.14343/) (2025)
- [Anatomica: Localized Control over Geometric and Topological Properties for Anatomical Diffusion Models](https://mlsanity.com/paper/2511.20587/) (2025)
- [Enhanced Landmark Detection Model in Pelvic Fluoroscopy using 2D/3D Registration Loss](https://mlsanity.com/paper/2511.21575/) (2025)
- [3D Path Planning for Robot-assisted Vertebroplasty from Arbitrary Bi-plane X-ray via Differentiable Rendering](https://mlsanity.com/paper/2512.05803/) (2025)
- [3-D Representations for Hyperspectral Flame Tomography](https://mlsanity.com/paper/2603.27832/) (2026)
- [Electromagnetic Navigation for Femoral Osteotomy Using High-Accuracy X-ray-to-CT Registration](https://mlsanity.com/paper/2606.03893/) (2026)
- [Patient-Specific Articulated Digital Twins from a Single Full-Body CT Scan](https://mlsanity.com/paper/2607.02156/) (2026)
- [Learning Dense 2D-3D Correspondence for X-ray-to-CT Registration of Knee Bones](https://mlsanity.com/paper/2607.22803/) (2026)
- [Patient-Agnostic Synthetic Pretraining for Efficient Patient-Specific Intraoperative 2D/3D Registration](https://mlsanity.com/paper/2607.23343/) (2026)
- [GeoPose: Patient-agnostic CTA-to-DSA registration through projection-space calibration](https://mlsanity.com/paper/2608.16600/) (2026)

## Abstract (arXiv)

Surgical decisions are informed by aligning rapid portable 2D intraoperative images (e.g., X-rays) to a high-fidelity 3D preoperative reference scan (e.g., CT). 2D/3D image registration often fails in practice: conventional optimization methods are prohibitively slow and susceptible to local minima, while neural networks trained on small datasets fail on new patients or require impractical landmark supervision. We present DiffPose, a self-supervised approach that leverages patient-specific simulation and differentiable physics-based rendering to achieve accurate 2D/3D registration without relying on manually labeled data. Preoperatively, a CNN is trained to regress the pose of a randomly oriented synthetic X-ray rendered from the preoperative CT. The CNN then initializes rapid intraoperative test-time optimization that uses the differentiable X-ray renderer to refine the solution. Our work further proposes several geometrically principled methods for sampling camera poses from SE(3), for sparse differentiable rendering, and for driving registration in the tangent space mathfrakse(3) with geodesic and multiscale locality-sensitive losses. DiffPose achieves sub-millimeter accuracy across surgical datasets at intraoperative speeds, improving upon existing unsupervised methods by an order of magnitude and even outperforming supervised baselines. Our code is available at https://github.com/eigenvivek/DiffPose.