# Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration

arXiv: https://arxiv.org/abs/2206.08903 · HTML: https://mlsanity.com/paper/2206.08903/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 65 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 16 of 16)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery](https://mlsanity.com/paper/1705.08260/) (2017)
- [Deep Learning and Conditional Random Fields-based Depth Estimation and Topographical Reconstruction from Conventional Endoscopy](https://mlsanity.com/paper/1710.11216/) (2017)
- [Automatic Annotation of Hip Anatomy in Fluoroscopy for Robust and Efficient 2D/3D Registration](https://mlsanity.com/paper/1911.07042/) (2019)
- [SLAM Endoscopy enhanced by adversarial depth prediction](https://mlsanity.com/paper/1907.00283/) (2019)
- [SERV-CT: A disparity dataset from CT for validation of endoscopic 3D reconstruction](https://mlsanity.com/paper/2012.11779/) (2020)
- [Bimodal Camera Pose Prediction for Endoscopy](https://mlsanity.com/paper/2204.04968/) (2022)
- [Detecting Deficient Coverage in Colonoscopies](https://mlsanity.com/paper/2001.08589/) (2020)
- [Deep Learning with Cinematic Rendering: Fine-Tuning Deep Neural Networks Using Photorealistic Medical Images](https://mlsanity.com/paper/1805.08400/) (2018)
- [Motion-based Camera Localization System in Colonoscopy Videos](https://mlsanity.com/paper/2012.01690/) (2020)
- [Rethinking Monocular Depth Estimation with Adversarial Training](https://mlsanity.com/paper/1808.07528/) (2018)
- [Multimodal Densenet](https://mlsanity.com/paper/1811.07407/) (2018)
- [GANPOP: Generative Adversarial Network Prediction of Optical Properties from Single Snapshot Wide-field Images](https://mlsanity.com/paper/1906.05360/) (2019)

## Built on by (most cited first, 30 of 80)

- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting](https://mlsanity.com/paper/2403.15124/) (2024)
- [LightNeuS: Neural Surface Reconstruction in Endoscopy using Illumination Decline](https://mlsanity.com/paper/2309.02777/) (2023)
- [SimCol3D -- 3D Reconstruction during Colonoscopy Challenge](https://mlsanity.com/paper/2307.11261/) (2023)
- [Gaussian Pancakes: Geometrically-Regularized 3D Gaussian Splatting for Realistic Endoscopic Reconstruction](https://mlsanity.com/paper/2404.06128/) (2024)
- [LightDepth: Single-View Depth Self-Supervision from Illumination Decline](https://mlsanity.com/paper/2308.10525/) (2023)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video](https://mlsanity.com/paper/2504.03198/) (2025)
- [Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping](https://mlsanity.com/paper/2309.16782/) (2023)
- [C3VDv2 -- Colonoscopy 3D video dataset with enhanced realism](https://mlsanity.com/paper/2506.24074/) (2025)
- [Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos](https://mlsanity.com/paper/2403.17915/) (2024)
- [Depth Anything in Medical Images: A Comparative Study](https://mlsanity.com/paper/2401.16600/) (2024)
- [EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels](https://mlsanity.com/paper/2409.05442/) (2024)
- [CudaSIFT-SLAM: multiple-map visual SLAM for full procedure mapping in real human endoscopy](https://mlsanity.com/paper/2405.16932/) (2024)
- [Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping](https://mlsanity.com/paper/2501.19319/) (2025)
- [EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splatting](https://mlsanity.com/paper/2506.21420/) (2025)
- [Register Any Point: Scaling 3D Point Cloud Registration by Flow Matching](https://mlsanity.com/paper/2512.01850/) (2025)
- [A Surface-normal Based Neural Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2303.07264/) (2023)
- [Multi-task learning with cross-task consistency for improved depth estimation in colonoscopy](https://mlsanity.com/paper/2311.18664/) (2023)
- [Building Rome with Convex Optimization](https://mlsanity.com/paper/2502.04640/) (2025)
- [A Review of 3D Reconstruction Techniques for Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2408.04426/) (2024)
- [SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data](https://mlsanity.com/paper/2412.16078/) (2024)
- [Structure-preserving Image Translation for Depth Estimation in Colonoscopy Video](https://mlsanity.com/paper/2408.10153/) (2024)
- [Toward Medical Deepfake Detection: A Comprehensive Dataset and Novel Method](https://mlsanity.com/paper/2509.15711/) (2025)
- [ColonMapper: topological mapping and localization for colonoscopy](https://mlsanity.com/paper/2305.05546/) (2023)
- [A Review on Organ Deformation Modeling Approaches for Reliable Surgical Navigation using Augmented Reality](https://mlsanity.com/paper/2408.02713/) (2024)
- [Motif Channel Opened in a White-Box: Stereo Matching via Motif Correlation Graph](https://mlsanity.com/paper/2411.12426/) (2024)
- [PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy](https://mlsanity.com/paper/2411.12510/) (2024)
- [NFL-BA: Near-Field Light Bundle Adjustment for SLAM in Dynamic Lighting](https://mlsanity.com/paper/2412.13176/) (2024)
- [From Pixels to Polygons: A Survey of Deep Learning Approaches for Medical Image-to-Mesh Reconstruction](https://mlsanity.com/paper/2505.03599/) (2025)

## Abstract (arXiv)

Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at durr.jhu.edu/C3VD.