# From Monocular Vision to Autonomous Action: Guiding Tumor Resection via 3D Reconstruction

arXiv: https://arxiv.org/abs/2503.16263 · HTML: https://mlsanity.com/paper/2503.16263/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 37 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Builds on (most cited first, 13 of 13)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [From Coarse to Fine: Robust Hierarchical Localization at Large Scale](https://mlsanity.com/paper/1812.03506/) (2018)
- [DISK: Learning local features with policy gradient](https://mlsanity.com/paper/2006.13566/) (2020)
- [ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation](https://mlsanity.com/paper/2304.03608/) (2023)
- [Reconstructing Sinus Anatomy from Endoscopic Video -- Towards a Radiation-free Approach for Quantitative Longitudinal Assessment](https://mlsanity.com/paper/2003.08502/) (2020)
- [Extremely Dense Point Correspondences using a Learned Feature Descriptor](https://mlsanity.com/paper/2003.00619/) (2020)
- [COSST: Multi-organ Segmentation with Partially Labeled Datasets Using Comprehensive Supervisions and Self-training](https://mlsanity.com/paper/2304.14030/) (2023)
- [Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot](https://mlsanity.com/paper/2002.06302/) (2020)
- [Autonomous Vision-Guided Resection of Central Airway Obstruction](https://mlsanity.com/paper/2502.18586/) (2025)

## Built on by (most cited first, 10 of 10)

- [PERSEUS: Perception with Semantic Endoscopic Understanding and SLAM](https://mlsanity.com/paper/2509.13541/) (2025)
- [A Survey of Medical Point Cloud Shape Learning: Registration, Reconstruction and Variation](https://mlsanity.com/paper/2508.03057/) (2025)
- [Monocular absolute depth estimation from endoscopy via domain-invariant feature learning and latent consistency](https://mlsanity.com/paper/2511.02247/) (2025)
- [SCARED-C: Corrected Camera Poses for Endoscopic Depth Estimation](https://mlsanity.com/paper/2605.16628/) (2026)
- [On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training](https://mlsanity.com/paper/2601.18929/) (2026)
- [Automated Assessment of Kidney Ureteroscopy Exploration for Training](https://mlsanity.com/paper/2602.15988/) (2026)
- [DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models](https://mlsanity.com/paper/2609.04555/) (2026)
- [Occupancy Network-Guided Autonomous Robotic Partial Nephrectomy](https://mlsanity.com/paper/2609.16186/) (2026)
- [Differentiable Mesh State Estimation via Factor Graph Inference for Deformable Object Reconstruction](https://mlsanity.com/paper/2609.16686/) (2026)
- [RAUL: Reference-Assisted Ureteroscopy Localization for Skill Assessment](https://mlsanity.com/paper/2609.19236/) (2026)

## Abstract (arXiv)

Surgical automation requires precise guidance and understanding of the scene. Current methods in the literature rely on bulky depth cameras to create maps of the anatomy, however this does not translate well to space-limited clinical applications. Monocular cameras are small and allow minimally invasive surgeries in tight spaces but additional processing is required to generate 3D scene understanding. We propose a 3D mapping pipeline that uses only RGB images to create segmented point clouds of the target anatomy. To ensure the most precise reconstruction, we compare different structure from motion algorithms' performance on mapping the central airway obstructions, and test the pipeline on a downstream task of tumor resection. In several metrics, including post-procedure tissue model evaluation, our pipeline performs comparably to RGB-D cameras and, in some cases, even surpasses their performance. These promising results demonstrate that automation guidance can be achieved in minimally invasive procedures with monocular cameras. This study is a step toward the complete autonomy of surgical robots.