# Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision

arXiv: https://arxiv.org/abs/2508.09681 · HTML: https://mlsanity.com/paper/2508.09681/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 13 Aug 2025.

## Builds on (most cited first, 26 of 26)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [NerfAcc: Efficient Sampling Accelerates NeRFs](https://mlsanity.com/paper/2305.04966/) (2023)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [MFT: Long-Term Tracking of Every Pixel](https://mlsanity.com/paper/2305.12998/) (2023)
- [Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods](https://mlsanity.com/paper/1902.07766/) (2019)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing](https://mlsanity.com/paper/1803.02009/) (2018)
- [Bimodal Camera Pose Prediction for Endoscopy](https://mlsanity.com/paper/2204.04968/) (2022)
- [Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction](https://mlsanity.com/paper/2407.02918/) (2024)
- [Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping](https://mlsanity.com/paper/2309.16782/) (2023)
- [SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Tracking](https://mlsanity.com/paper/2305.06477/) (2023)
- [Track Everything Everywhere Fast and Robustly](https://mlsanity.com/paper/2403.17931/) (2024)
- [Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery](https://mlsanity.com/paper/2110.03912/) (2021)
- [Tracking monocular camera pose and deformation for SLAM inside the human body](https://mlsanity.com/paper/2204.08309/) (2022)
- [Online 3D reconstruction and dense tracking in endoscopic videos](https://mlsanity.com/paper/2409.06037/) (2024)
- [Point Tracking in Surgery--The 2024 Surgical Tattoos in Infrared (STIR) Challenge](https://mlsanity.com/paper/2503.24306/) (2025)
- [Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting](https://mlsanity.com/paper/2407.14846/) (2024)
- [Tracking Everything in Robotic-Assisted Surgery](https://mlsanity.com/paper/2409.19821/) (2024)
- [3D endoscopic depth estimation using 3D surface-aware constraints](https://mlsanity.com/paper/2203.02131/) (2022)

## Abstract (arXiv)

We proposed a novel test-time optimisation (TTO) approach framed by a NeRF-based architecture for long-term 3D point tracking. Most current methods in point tracking struggle to obtain consistent motion or are limited to 2D motion. TTO approaches frame the solution for long-term tracking as optimising a function that aggregates correspondences from other specialised state-of-the-art methods. Unlike the state-of-the-art on TTO, we propose parametrising such a function with our new invertible Neural Radiance Field (InvNeRF) architecture to perform both 2D and 3D tracking in surgical scenarios. Our approach allows us to exploit the advantages of a rendering-based approach by supervising the reprojection of pixel correspondences. It adapts strategies from recent rendering-based methods to obtain a bidirectional deformable-canonical mapping, to efficiently handle a defined workspace, and to guide the rays' density. It also presents our multi-scale HexPlanes for fast inference and a new algorithm for efficient pixel sampling and convergence criteria. We present results in the STIR and SCARE datasets, for evaluating point tracking and testing the integration of kinematic data in our pipeline, respectively. In 2D point tracking, our approach surpasses the precision and accuracy of the TTO state-of-the-art methods by nearly 50% on average precision, while competing with other approaches. In 3D point tracking, this is the first TTO approach, surpassing feed-forward methods while incorporating the benefits of a deformable NeRF-based reconstruction.