# FLex: Joint Pose and Dynamic Radiance Fields Optimization for Stereo Endoscopic Videos

arXiv: https://arxiv.org/abs/2403.12198 · HTML: https://mlsanity.com/paper/2403.12198/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 20 of 20)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [Learning How To Robustly Estimate Camera Pose in Endoscopic Videos](https://mlsanity.com/paper/2304.08023/) (2023)
- [Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues](https://mlsanity.com/paper/2305.19906/) (2023)
- [Reconstructing Sinus Anatomy from Endoscopic Video -- Towards a Radiation-free Approach for Quantitative Longitudinal Assessment](https://mlsanity.com/paper/2003.08502/) (2020)
- [Dynamic Reconstruction of Deformable Soft-tissue with Stereo Scope in Minimal Invasive Surgery](https://mlsanity.com/paper/2003.10867/) (2020)
- [Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane](https://mlsanity.com/paper/2312.15253/) (2023)
- [Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario](https://mlsanity.com/paper/2201.00383/) (2022)
- [BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields](https://mlsanity.com/paper/2309.15329/) (2023)

## Built on by (most cited first, 3 of 3)

- [Free-DyGS: Camera-Pose-Free Scene Reconstruction for Dynamic Surgical Videos with Gaussian Splatting](https://mlsanity.com/paper/2409.01003/) (2024)
- [Tracking-Aware Deformation Field Estimation for Non-rigid 3D Reconstruction in Robotic Surgeries](https://mlsanity.com/paper/2503.02558/) (2025)
- [Monocular Endoscopic Tissue 3D Reconstruction with Multi-Level Geometry Regularization](https://mlsanity.com/paper/2602.20718/) (2026)

## Abstract (arXiv)

Reconstruction of endoscopic scenes is an important asset for various medical applications, from post-surgery analysis to educational training. Neural rendering has recently shown promising results in endoscopic reconstruction with deforming tissue. However, the setup has been restricted to a static endoscope, limited deformation, or required an external tracking device to retrieve camera pose information of the endoscopic camera. With FLex we adress the challenging setup of a moving endoscope within a highly dynamic environment of deforming tissue. We propose an implicit scene separation into multiple overlapping 4D neural radiance fields (NeRFs) and a progressive optimization scheme jointly optimizing for reconstruction and camera poses from scratch. This improves the ease-of-use and allows to scale reconstruction capabilities in time to process surgical videos of 5,000 frames and more; an improvement of more than ten times compared to the state of the art while being agnostic to external tracking information. Extensive evaluations on the StereoMIS dataset show that FLex significantly improves the quality of novel view synthesis while maintaining competitive pose accuracy.