# Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation

arXiv: https://arxiv.org/abs/2003.12473 · HTML: https://mlsanity.com/paper/2003.12473/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2020.

## Built on by (most cited first, 29 of 29)

- [Colonoscopy Polyp Detection and Classification: Dataset Creation and Comparative Evaluations](https://mlsanity.com/paper/2104.10824/) (2021)
- [Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data](https://mlsanity.com/paper/2103.17204/) (2021)
- [Bimodal Camera Pose Prediction for Endoscopy](https://mlsanity.com/paper/2204.04968/) (2022)
- [SimCol3D -- 3D Reconstruction during Colonoscopy Challenge](https://mlsanity.com/paper/2307.11261/) (2023)
- [LightDepth: Single-View Depth Self-Supervision from Illumination Decline](https://mlsanity.com/paper/2308.10525/) (2023)
- [Robustness via Uncertainty-aware Cycle Consistency](https://mlsanity.com/paper/2110.12467/) (2021)
- [Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging](https://mlsanity.com/paper/2107.09543/) (2021)
- [EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels](https://mlsanity.com/paper/2409.05442/) (2024)
- [BronchoTrack: Airway Lumen Tracking for Branch-Level Bronchoscopic Localization](https://mlsanity.com/paper/2402.12763/) (2024)
- [A Surface-normal Based Neural Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2303.07264/) (2023)
- [ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2111.10371/) (2021)
- [GAN Inversion for Data Augmentation to Improve Colonoscopy Lesion Classification](https://mlsanity.com/paper/2205.02840/) (2022)
- [DD-VNB: A Depth-based Dual-Loop Framework for Real-time Visually Navigated Bronchoscopy](https://mlsanity.com/paper/2403.01683/) (2024)
- [FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos](https://mlsanity.com/paper/2106.12522/) (2021)
- [Task-guided Domain Gap Reduction for Monocular Depth Prediction in Endoscopy](https://mlsanity.com/paper/2310.01663/) (2023)
- [CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy](https://mlsanity.com/paper/2206.14951/) (2022)
- [PCT-CycleGAN: Paired Complementary Temporal Cycle-Consistent Adversarial Networks for Radar-Based Precipitation Nowcasting](https://mlsanity.com/paper/2211.15046/) (2022)
- [Semi-supervised Bladder Tissue Classification in Multi-Domain Endoscopic Images](https://mlsanity.com/paper/2212.11375/) (2022)
- [MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production](https://mlsanity.com/paper/2407.12842/) (2024)
- [ColonCrafter: A Depth Estimation Model for Colonoscopy Videos Using Diffusion Priors](https://mlsanity.com/paper/2509.13525/) (2025)
- [Visualizing Missing Surfaces In Colonoscopy Videos using Shared Latent Space Representations](https://mlsanity.com/paper/2101.07280/) (2021)
- [Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images](https://mlsanity.com/paper/2311.05021/) (2023)
- [NeuRegenerate: A Framework for Visualizing Neurodegeneration](https://mlsanity.com/paper/2202.01115/) (2022)
- [SoftEnNet: Symbiotic Monocular Depth Estimation and Lumen Segmentation for Colonoscopy Endorobots](https://mlsanity.com/paper/2301.08157/) (2023)
- [RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches](https://mlsanity.com/paper/2310.00868/) (2023)
- [Sim2Real within 5 Minutes: Efficient Domain Transfer with Stylized Gaussian Splatting for Endoscopic Images](https://mlsanity.com/paper/2403.10860/) (2024)
- [Multi-Exposure Image Fusion via Distilled 3D LUT Grid with Editable Mode](https://mlsanity.com/paper/2412.13749/) (2024)
- [Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation](https://mlsanity.com/paper/2602.21740/) (2026)
- [EndoSERV: A Vision-based Endoluminal Robot Navigation System](https://mlsanity.com/paper/2603.08324/) (2026)

## Abstract (arXiv)

Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. Code/models: https://github.com/nadeemlab/CEP.