# A Surface-normal Based Neural Framework for Colonoscopy Reconstruction

arXiv: https://arxiv.org/abs/2303.07264 · HTML: https://mlsanity.com/paper/2303.07264/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training](https://mlsanity.com/paper/1711.06606/) (2017)
- [LEGO: Learning Edge with Geometry all at Once by Watching Videos](https://mlsanity.com/paper/1803.05648/) (2018)
- [Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods](https://mlsanity.com/paper/1902.07766/) (2019)
- [Shape and Material Capture at Home](https://mlsanity.com/paper/2104.06397/) (2021)
- [StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation](https://mlsanity.com/paper/2108.08574/) (2021)
- [Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation](https://mlsanity.com/paper/2003.12473/) (2020)
- [Reconstructing Sinus Anatomy from Endoscopic Video -- Towards a Radiation-free Approach for Quantitative Longitudinal Assessment](https://mlsanity.com/paper/2003.08502/) (2020)
- [Fast Light-Weight Near-Field Photometric Stereo](https://mlsanity.com/paper/2203.16515/) (2022)
- [Lighting Enhancement Aids Reconstruction of Colonoscopic Surfaces](https://mlsanity.com/paper/2103.10310/) (2021)

## Built on by (most cited first, 8 of 8)

- [C3VDv2 -- Colonoscopy 3D video dataset with enhanced realism](https://mlsanity.com/paper/2506.24074/) (2025)
- [Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos](https://mlsanity.com/paper/2403.17915/) (2024)
- [Structure-preserving Image Translation for Depth Estimation in Colonoscopy Video](https://mlsanity.com/paper/2408.10153/) (2024)
- [NFL-BA: Near-Field Light Bundle Adjustment for SLAM in Dynamic Lighting](https://mlsanity.com/paper/2412.13176/) (2024)
- [SHADeS: Self-supervised Monocular Depth Estimation Through Non-Lambertian Image Decomposition](https://mlsanity.com/paper/2502.12994/) (2025)
- [ColonNeRF: High-Fidelity Neural Reconstruction of Long Colonoscopy](https://mlsanity.com/paper/2312.02015/) (2023)
- [Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation](https://mlsanity.com/paper/2602.21740/) (2026)
- [Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation](https://mlsanity.com/paper/2603.15374/) (2026)

## Abstract (arXiv)

Reconstructing a 3D surface from colonoscopy video is challenging due to illumination and reflectivity variation in the video frame that can cause defective shape predictions. Aiming to overcome this challenge, we utilize the characteristics of surface normal vectors and develop a two-step neural framework that significantly improves the colonoscopy reconstruction quality. The normal-based depth initialization network trained with self-supervised normal consistency loss provides depth map initialization to the normal-depth refinement module, which utilizes the relationship between illumination and surface normals to refine the frame-wise normal and depth predictions recursively. Our framework's depth accuracy performance on phantom colonoscopy data demonstrates the value of exploiting the surface normals in colonoscopy reconstruction, especially on en face views. Due to its low depth error, the prediction result from our framework will require limited post-processing to be clinically applicable for real-time colonoscopy reconstruction.