# SAGE: SLAM with Appearance and Geometry Prior for Endoscopy

arXiv: https://arxiv.org/abs/2202.09487 · HTML: https://mlsanity.com/paper/2202.09487/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 19 Feb 2022.

## Builds on (most cited first, 25 of 25)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [TEASER: Fast and Certifiable Point Cloud Registration](https://mlsanity.com/paper/2001.07715/) (2020)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM](https://mlsanity.com/paper/1804.00874/) (2018)
- [DeepFactors: Real-Time Probabilistic Dense Monocular SLAM](https://mlsanity.com/paper/2001.05049/) (2020)
- [DeepSFM: Structure From Motion Via Deep Bundle Adjustment](https://mlsanity.com/paper/1912.09697/) (2019)
- [Visual Odometry Revisited: What Should Be Learnt?](https://mlsanity.com/paper/1909.09803/) (2019)
- [Pseudo RGB-D for Self-Improving Monocular SLAM and Depth Prediction](https://mlsanity.com/paper/2004.10681/) (2020)
- [DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences](https://mlsanity.com/paper/1908.08918/) (2019)
- [MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing](https://mlsanity.com/paper/1803.02009/) (2018)
- [Extremely Dense Point Correspondences using a Learned Feature Descriptor](https://mlsanity.com/paper/2003.00619/) (2020)
- [Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots](https://mlsanity.com/paper/1803.01047/) (2018)
- [SLAM based Quasi Dense Reconstruction For Minimally Invasive Surgery Scenes](https://mlsanity.com/paper/1705.09107/) (2017)
- [DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image Translation](https://mlsanity.com/paper/2005.03995/) (2020)
- [A Non-Rigid Map Fusion-Based RGB-Depth SLAM Method for Endoscopic Capsule Robots](https://mlsanity.com/paper/1705.05444/) (2017)
- [An observable time series based SLAM algorithm for deforming environment](https://mlsanity.com/paper/1906.08563/) (2019)

## Built on by (most cited first, 28 of 28)

- [Learning How To Robustly Estimate Camera Pose in Endoscopic Videos](https://mlsanity.com/paper/2304.08023/) (2023)
- [Tracking and Mapping in Medical Computer Vision: A Review](https://mlsanity.com/paper/2310.11475/) (2023)
- [EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting](https://mlsanity.com/paper/2403.15124/) (2024)
- [LightNeuS: Neural Surface Reconstruction in Endoscopy using Illumination Decline](https://mlsanity.com/paper/2309.02777/) (2023)
- [CudaSIFT-SLAM: multiple-map visual SLAM for full procedure mapping in real human endoscopy](https://mlsanity.com/paper/2405.16932/) (2024)
- [Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping](https://mlsanity.com/paper/2501.19319/) (2025)
- [EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splatting](https://mlsanity.com/paper/2506.21420/) (2025)
- [Topological SLAM in colonoscopies leveraging deep features and topological priors](https://mlsanity.com/paper/2409.16806/) (2024)
- [ColonMapper: topological mapping and localization for colonoscopy](https://mlsanity.com/paper/2305.05546/) (2023)
- [Detecting the Sensing Area of A Laparoscopic Probe in Minimally Invasive Cancer Surgery](https://mlsanity.com/paper/2307.03662/) (2023)
- [RIDE: Self-Supervised Learning of Rotation-Equivariant Keypoint Detection and Invariant Description for Endoscopy](https://mlsanity.com/paper/2309.09563/) (2023)
- [BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields](https://mlsanity.com/paper/2309.15329/) (2023)
- [BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applications](https://mlsanity.com/paper/2408.03078/) (2024)
- [PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy](https://mlsanity.com/paper/2411.12510/) (2024)
- [Autonomous Vision-Guided Resection of Central Airway Obstruction](https://mlsanity.com/paper/2502.18586/) (2025)
- [TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery](https://mlsanity.com/paper/2212.14131/) (2022)
- [3D Densification for Multi-Map Monocular VSLAM in Endoscopy](https://mlsanity.com/paper/2503.14346/) (2025)
- [PERSEUS: Perception with Semantic Endoscopic Understanding and SLAM](https://mlsanity.com/paper/2509.13541/) (2025)
- [A Quantitative Evaluation of Dense 3D Reconstruction of Sinus Anatomy from Monocular Endoscopic Video](https://mlsanity.com/paper/2310.14364/) (2023)
- [ColonNeRF: High-Fidelity Neural Reconstruction of Long Colonoscopy](https://mlsanity.com/paper/2312.02015/) (2023)
- [BDIS-SLAM: A lightweight CPU-based dense stereo SLAM for surgery](https://mlsanity.com/paper/2312.15679/) (2023)
- [Registering Neural 4D Gaussians for Endoscopic Surgery](https://mlsanity.com/paper/2407.20213/) (2024)
- [An Endoscopic Chisel: Intraoperative Imaging Carves 3D Anatomical Models](https://mlsanity.com/paper/2402.11840/) (2024)
- [EndoPBR: Material and Lighting Estimation for Photorealistic Surgical Simulations via Physically-based Rendering](https://mlsanity.com/paper/2502.20669/) (2025)
- [SuperPoint-E: local features for 3D reconstruction via tracking adaptation in endoscopy](https://mlsanity.com/paper/2602.04108/) (2026)
- [Investigating a Policy-Based Formulation for Endoscopic Camera Pose Recovery](https://mlsanity.com/paper/2603.20045/) (2026)
- [C3VD-DEFCOL: A Deformable Colonoscopy Dataset with Time-Resolved 3D Ground Truth and Realistic Appearance](https://mlsanity.com/paper/2606.07891/) (2026)
- [EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition](https://mlsanity.com/paper/2608.08949/) (2026)

## Abstract (arXiv)

In endoscopy, many applications (e.g., surgical navigation) would benefit from a real-time method that can simultaneously track the endoscope and reconstruct the dense 3D geometry of the observed anatomy from a monocular endoscopic video. To this end, we develop a Simultaneous Localization and Mapping system by combining the learning-based appearance and optimizable geometry priors and factor graph optimization. The appearance and geometry priors are explicitly learned in an end-to-end differentiable training pipeline to master the task of pair-wise image alignment, one of the core components of the SLAM system. In our experiments, the proposed SLAM system is shown to robustly handle the challenges of texture scarceness and illumination variation that are commonly seen in endoscopy. The system generalizes well to unseen endoscopes and subjects and performs favorably compared with a state-of-the-art feature-based SLAM system. The code repository is available at https://github.com/lppllppl920/SAGE-SLAM.git.