# Intraoperative Registration by Cross-Modal Inverse Neural Rendering

arXiv: https://arxiv.org/abs/2409.11983 · HTML: https://mlsanity.com/paper/2409.11983/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Style Transfer by Relaxed Optimal Transport and Self-Similarity](https://mlsanity.com/paper/1904.12785/) (2019)
- [Photorealistic Style Transfer via Wavelet Transforms](https://mlsanity.com/paper/1903.09760/) (2019)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [Stylizing 3D Scene via Implicit Representation and HyperNetwork](https://mlsanity.com/paper/2105.13016/) (2021)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [Parallel Inversion of Neural Radiance Fields for Robust Pose Estimation](https://mlsanity.com/paper/2210.10108/) (2022)
- [Learning Expected Appearances for Intraoperative Registration during Neurosurgery](https://mlsanity.com/paper/2310.01735/) (2023)

## Built on by (most cited first, 3 of 3)

- [A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration](https://mlsanity.com/paper/2507.18551/) (2025)
- [Surgical Neural Radiance Fields from One Image](https://mlsanity.com/paper/2507.00969/) (2025)
- [Implicit Deformable Medical Image Registration with Learnable Kernels](https://mlsanity.com/paper/2506.02150/) (2025)

## Abstract (arXiv)

We present in this paper a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. This disentanglement is achieved by controlling a Neural Radiance Field's appearance with a multi-style hypernetwork. Once trained, the implicit neural representation serves as a differentiable rendering engine, which can be used to estimate the surgical camera pose by minimizing the dissimilarity between its rendered images and the target intraoperative image. We tested our method on retrospective patients' data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration. Code and additional resources can be found at https://maxfehrentz.github.io/style-ngp/.