# Depth estimation of endoscopy using sim-to-real transfer

arXiv: https://arxiv.org/abs/2112.13595 · HTML: https://mlsanity.com/paper/2112.13595/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 27 Dec 2021.

## Builds on (most cited first, 9 of 9)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real](https://mlsanity.com/paper/2006.09001/) (2020)
- [Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots](https://mlsanity.com/paper/1803.01047/) (2018)

## Built on by (most cited first, 6 of 6)

- [Depth Anything in Medical Images: A Comparative Study](https://mlsanity.com/paper/2401.16600/) (2024)
- [Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study](https://mlsanity.com/paper/2206.15086/) (2022)
- [PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation](https://mlsanity.com/paper/2504.17067/) (2025)
- [Generalizing monocular colonoscopy image depth estimation by uncertainty-based global and local fusion network](https://mlsanity.com/paper/2409.15006/) (2024)
- [Multi-Modal Monocular Endoscopic Depth and Pose Estimation with Edge-Guided Self-Supervision](https://mlsanity.com/paper/2602.17785/) (2026)
- [Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation](https://mlsanity.com/paper/2602.21740/) (2026)

## Abstract (arXiv)

In order to use the navigation system effectively, distance information sensors such as depth sensors are essential. Since depth sensors are difficult to use in endoscopy, many groups propose a method using convolutional neural networks. In this paper, the ground truth of the depth image and the endoscopy image is generated through endoscopy simulation using the colon model segmented by CT colonography. Photo-realistic simulation images can be created using a sim-to-real approach using cycleGAN for endoscopy images. By training the generated dataset, we propose a quantitative endoscopy depth estimation network. The proposed method represents a better-evaluated score than the existing unsupervised training-based results.