# PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation

arXiv: https://arxiv.org/abs/2504.17067 · HTML: https://mlsanity.com/paper/2504.17067/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 20 of 20)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [EndoMapper dataset of complete calibrated endoscopy procedures](https://mlsanity.com/paper/2204.14240/) (2022)
- [ArSDM: Colonoscopy Images Synthesis with Adaptive Refinement Semantic Diffusion Models](https://mlsanity.com/paper/2309.01111/) (2023)
- [Atlantis: Enabling Underwater Depth Estimation with Stable Diffusion](https://mlsanity.com/paper/2312.12471/) (2023)
- [ZeST: Zero-Shot Material Transfer from a Single Image](https://mlsanity.com/paper/2404.06425/) (2024)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [SimCol3D -- 3D Reconstruction during Colonoscopy Challenge](https://mlsanity.com/paper/2307.11261/) (2023)
- [LightDepth: Single-View Depth Self-Supervision from Illumination Decline](https://mlsanity.com/paper/2308.10525/) (2023)
- [Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos](https://mlsanity.com/paper/2403.17915/) (2024)
- [Fast Light-Weight Near-Field Photometric Stereo](https://mlsanity.com/paper/2203.16515/) (2022)
- [Depth estimation of endoscopy using sim-to-real transfer](https://mlsanity.com/paper/2112.13595/) (2021)
- [Structure-preserving Image Translation for Depth Estimation in Colonoscopy Video](https://mlsanity.com/paper/2408.10153/) (2024)
- [CCIS-Diff: A Generative Model with Stable Diffusion Prior for Controlled Colonoscopy Image Synthesis](https://mlsanity.com/paper/2411.12198/) (2024)
- [SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Models](https://mlsanity.com/paper/2408.09822/) (2024)
- [NFL-BA: Near-Field Light Bundle Adjustment for SLAM in Dynamic Lighting](https://mlsanity.com/paper/2412.13176/) (2024)
- [ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation](https://mlsanity.com/paper/2407.16508/) (2024)

## Built on by (most cited first, 2 of 2)

- [ColonCrafter: A Depth Estimation Model for Colonoscopy Videos Using Diffusion Priors](https://mlsanity.com/paper/2509.13525/) (2025)
- [Monocular absolute depth estimation from endoscopy via domain-invariant feature learning and latent consistency](https://mlsanity.com/paper/2511.02247/) (2025)

## Abstract (arXiv)

Accurate depth estimation enhances endoscopy navigation and diagnostics, but obtaining ground-truth depth in clinical settings is challenging. Synthetic datasets are often used for training, yet the domain gap limits generalization to real data. We propose a novel image-to-image translation framework that preserves structure while generating realistic textures from clinical data. Our key innovation integrates Stable Diffusion with ControlNet, conditioned on a latent representation extracted from a Per-Pixel Shading (PPS) map. PPS captures surface lighting effects, providing a stronger structural constraint than depth maps. Experiments show our approach produces more realistic translations and improves depth estimation over GAN-based MI-CycleGAN. Our code is publicly accessible at https://github.com/anaxqx/PPS-Ctrl.