# Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery arXiv: https://arxiv.org/abs/2110.03912 ยท HTML: https://mlsanity.com/paper/2110.03912/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 18 of its 43 references that are in the map. - First posted on arXiv 8 Oct 2021. ## Builds on (most cited first, 18 of 18) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016) - [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015) - [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016) - [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018) - [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016) - [AANet: Adaptive Aggregation Network for Efficient Stereo Matching](https://mlsanity.com/paper/2004.09548/) (2020) - [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020) - [Geometry-Aware Learning of Maps for Camera Localization](https://mlsanity.com/paper/1712.03342/) (2017) - [Hierarchical Deep Stereo Matching on High-resolution Images](https://mlsanity.com/paper/1912.06704/) (2019) - [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021) - [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021) - [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021) - [Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization](https://mlsanity.com/paper/1809.01019/) (2018) - [Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images](https://mlsanity.com/paper/2107.04644/) (2021) - [Real-time Dense Reconstruction of Tissue Surface from Stereo Optical Video](https://mlsanity.com/paper/2007.12623/) (2020) - [Adversarial Domain Feature Adaptation for Bronchoscopic Depth Estimation](https://mlsanity.com/paper/2109.11798/) (2021) - [Self-supervised Depth Estimation to Regularise Semantic Segmentation in Knee Arthroscopy](https://mlsanity.com/paper/2007.02361/) (2020) ## Built on by (most cited first, 11 of 11) - [Learning How To Robustly Estimate Camera Pose in Endoscopic Videos](https://mlsanity.com/paper/2304.08023/) (2023) - [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023) - [EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting](https://mlsanity.com/paper/2403.15124/) (2024) - [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024) - [Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling](https://mlsanity.com/paper/2408.07266/) (2024) - [Robotic Surgery Remote Mentoring via AR with 3D Scene Streaming and Hand Interaction](https://mlsanity.com/paper/2204.04377/) (2022) - [A Disparity Refinement Framework for Learning-based Stereo Matching Methods in Cross-domain Setting for Laparoscopic Images](https://mlsanity.com/paper/2302.02294/) (2023) - [3 Dimensional Dense Reconstruction: A Review of Algorithms and Dataset](https://mlsanity.com/paper/2304.09371/) (2023) - [Monocular Depth Guided Occlusion-Aware Disparity Refinement via Semi-supervised Learning in Laparoscopic Images](https://mlsanity.com/paper/2505.08178/) (2025) - [Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision](https://mlsanity.com/paper/2508.09681/) (2025) - [TiS-TSL: Image-Label Supervised Surgical Video Stereo Matching via Time-Switchable Teacher-Student Learning](https://mlsanity.com/paper/2511.06817/) (2025) ## Abstract (arXiv) Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environment, we first fine-tune a learning-based stereoscopic depth perception method, which is robust to the texture-less and variant soft tissues, for depth estimation. Then, we develop a dense visual reconstruction algorithm to represent the scene by surfels, estimate the laparoscope poses and fuse the depth maps into a unified reference coordinate for tissue reconstruction. To estimate poses of new laparoscope views, we achieve a coarse-to-fine localization method, which incorporates our reconstructed 3D model. Results: We evaluate the reconstruction method and the localization module on three datasets, namely, the stereo correspondence and reconstruction of endoscopic data (SCARED), the ex-vivo phantom and tissue data collected with Universal Robot (UR) and Karl Storz Laparoscope, and the in-vivo DaVinci robotic surgery dataset, where the reconstructed 3D structures have rich details of surface texture with an accuracy error under 1.71 mm and the localization module can accurately track the laparoscope with only images as input. Conclusions: Experimental results demonstrate the superior performance of the proposed method in 3D anatomy reconstruction and laparoscopic localization. Significance: The proposed framework can be potentially extended to the current surgical navigation system.