# Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images

arXiv: https://arxiv.org/abs/2311.05021 · HTML: https://mlsanity.com/paper/2311.05021/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 77 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [High Quality Monocular Depth Estimation via Transfer Learning](https://mlsanity.com/paper/1812.11941/) (2018)
- [The Importance of Skip Connections in Biomedical Image Segmentation](https://mlsanity.com/paper/1608.04117/) (2016)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [Monocular Depth Estimation: A Survey](https://mlsanity.com/paper/1901.09402/) (2019)
- [Deep Learning and Conditional Random Fields-based Depth Estimation and Topographical Reconstruction from Conventional Endoscopy](https://mlsanity.com/paper/1710.11216/) (2017)
- [Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation](https://mlsanity.com/paper/2003.12473/) (2020)
- [VR-Caps: A Virtual Environment for Capsule Endoscopy](https://mlsanity.com/paper/2008.12949/) (2020)
- [SERV-CT: A disparity dataset from CT for validation of endoscopic 3D reconstruction](https://mlsanity.com/paper/2012.11779/) (2020)
- [Detecting Deficient Coverage in Colonoscopies](https://mlsanity.com/paper/2001.08589/) (2020)
- [Error Diagnosis of Deep Monocular Depth Estimation Models](https://mlsanity.com/paper/2112.05533/) (2021)
- [Real-time automatic polyp detection in colonoscopy using feature enhancement module and spatiotemporal similarity correlation unit](https://mlsanity.com/paper/2201.10079/) (2022)

## Built on by (most cited first, 2 of 2)

- [Endoscopic Depth Estimation Based on Deep Learning: A Survey](https://mlsanity.com/paper/2507.20881/) (2025)
- [V2-SfMLearner: Learning Monocular Depth and Ego-motion for Multimodal Wireless Capsule Endoscopy](https://mlsanity.com/paper/2412.17595/) (2024)

## Abstract (arXiv)

Colonoscopy is the choice procedure to diagnose colon and rectum cancer, from early detection of small precancerous lesions (polyps), to confirmation of malign masses. However, the high variability of the organ appearance and the complex shape of both the colon wall and structures of interest make this exploration difficult. Learned visuospatial and perceptual abilities mitigate technical limitations in clinical practice by proper estimation of the intestinal depth. This work introduces a novel methodology to estimate colon depth maps in single frames from monocular colonoscopy videos. The generated depth map is inferred from the shading variation of the colon wall with respect to the light source, as learned from a realistic synthetic database. Briefly, a classic convolutional neural network architecture is trained from scratch to estimate the depth map, improving sharp depth estimations in haustral folds and polyps by a custom loss function that minimizes the estimation error in edges and curvatures. The network was trained by a custom synthetic colonoscopy database herein constructed and released, composed of 248,400 frames (47 videos), with depth annotations at the level of pixels. This collection comprehends 5 subsets of videos with progressively higher levels of visual complexity. Evaluation of the depth estimation with the synthetic database reached a threshold accuracy of 95.65%, and a mean-RMSE of 0.451 cm, while a qualitative assessment with a real database showed consistent depth estimations, visually evaluated by the expert gastroenterologist coauthoring this paper. Finally, the method achieved competitive performance with respect to another state-of-the-art method using a public synthetic database and comparable results in a set of images with other five state-of-the-art methods.