# Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation

arXiv: https://arxiv.org/abs/2602.21740 · HTML: https://mlsanity.com/paper/2602.21740/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Builds on (most cited first, 14 of 14)

- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [Deep Learning and Conditional Random Fields-based Depth Estimation and Topographical Reconstruction from Conventional Endoscopy](https://mlsanity.com/paper/1710.11216/) (2017)
- [SurfelMeshing: Online Surfel-Based Mesh Reconstruction](https://mlsanity.com/paper/1810.00729/) (2018)
- [Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation](https://mlsanity.com/paper/2003.12473/) (2020)
- [Bimodal Camera Pose Prediction for Endoscopy](https://mlsanity.com/paper/2204.04968/) (2022)
- [LightDepth: Single-View Depth Self-Supervision from Illumination Decline](https://mlsanity.com/paper/2308.10525/) (2023)
- [Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos](https://mlsanity.com/paper/2403.17915/) (2024)
- [A Surface-normal Based Neural Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2303.07264/) (2023)
- [Multi-task learning with cross-task consistency for improved depth estimation in colonoscopy](https://mlsanity.com/paper/2311.18664/) (2023)
- [Depth estimation of endoscopy using sim-to-real transfer](https://mlsanity.com/paper/2112.13595/) (2021)
- [Structure-preserving Image Translation for Depth Estimation in Colonoscopy Video](https://mlsanity.com/paper/2408.10153/) (2024)
- [ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation](https://mlsanity.com/paper/2407.16508/) (2024)
- [Sim2Real in endoscopy segmentation with a novel structure aware image translation](https://mlsanity.com/paper/2505.02654/) (2025)

## Abstract (arXiv)

Monocular depth estimation (MDE) for colonoscopy is hampered by the domain gap between simulated and real-world images. Existing image-to-image translation methods, which use depth as a posterior constraint, often produce structural distortions and specular highlights by failing to balance realism with structure consistency. To address this, we propose a Structure-to-Image paradigm that transforms the depth map from a passive constraint into an active generative foundation. We are the first to introduce phase congruency to colonoscopic domain adaptation and design a cross-level structure constraint to co-optimize geometric structures and fine-grained details like vascular textures. In zero-shot evaluations conducted on a publicly available phantom dataset, the MDE model that was fine-tuned on our generated data achieved a maximum reduction of 44.18% in RMSE compared to competing methods. Our code is available at https://github.com/YyangJJuan/PC-S2I.git.