# VR-Caps: A Virtual Environment for Capsule Endoscopy

arXiv: https://arxiv.org/abs/2008.12949 · HTML: https://mlsanity.com/paper/2008.12949/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2020.

## Built on by (most cited first, 23 of 23)

- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [EndoMapper dataset of complete calibrated endoscopy procedures](https://mlsanity.com/paper/2204.14240/) (2022)
- [Physics-driven Synthetic Data Learning for Biomedical Magnetic Resonance](https://mlsanity.com/paper/2203.11178/) (2022)
- [LightDepth: Single-View Depth Self-Supervision from Illumination Decline](https://mlsanity.com/paper/2308.10525/) (2023)
- [Tracking monocular camera pose and deformation for SLAM inside the human body](https://mlsanity.com/paper/2204.08309/) (2022)
- [Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study](https://mlsanity.com/paper/2206.15086/) (2022)
- [FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos](https://mlsanity.com/paper/2106.12522/) (2021)
- [CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy](https://mlsanity.com/paper/2206.14951/) (2022)
- [Automatic Generation of Synthetic Colonoscopy Videos for Domain Randomization](https://mlsanity.com/paper/2205.10368/) (2022)
- [EndoMetric: Near-Light Monocular Metric Scale Estimation in Endoscopy](https://mlsanity.com/paper/2410.15065/) (2024)
- [Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images](https://mlsanity.com/paper/2311.05021/) (2023)
- [ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation](https://mlsanity.com/paper/2407.16508/) (2024)
- [RealSynCol: a high-fidelity synthetic colon dataset for 3D reconstruction applications](https://mlsanity.com/paper/2602.08397/) (2026)
- [Safe Navigation for Robotic Digestive Endoscopy via Human Intervention-based Reinforcement Learning](https://mlsanity.com/paper/2409.15688/) (2024)
- [Sim2Real in endoscopy segmentation with a novel structure aware image translation](https://mlsanity.com/paper/2505.02654/) (2025)
- [Endoscopic Depth Estimation Based on Deep Learning: A Survey](https://mlsanity.com/paper/2507.20881/) (2025)
- [4D Monocular Surgical Reconstruction under Arbitrary Camera Motions](https://mlsanity.com/paper/2602.17473/) (2026)
- [RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches](https://mlsanity.com/paper/2310.00868/) (2023)
- [3D Reconstruction of the Human Colon from Capsule Endoscope Video](https://mlsanity.com/paper/2407.15228/) (2024)
- [V2-SfMLearner: Learning Monocular Depth and Ego-motion for Multimodal Wireless Capsule Endoscopy](https://mlsanity.com/paper/2412.17595/) (2024)
- [eNCApsulate: NCA for Precision Diagnosis on Capsule Endoscopes](https://mlsanity.com/paper/2504.21562/) (2025)
- [CoGE: Sim-to-Real Online Geometric Estimation for Monocular Colonoscopy](https://mlsanity.com/paper/2605.13038/) (2026)
- [C3VD-DEFCOL: A Deformable Colonoscopy Dataset with Time-Resolved 3D Ground Truth and Realistic Appearance](https://mlsanity.com/paper/2606.07891/) (2026)

## Abstract (arXiv)

Current capsule endoscopes and next-generation robotic capsules for diagnosis and treatment of gastrointestinal diseases are complex cyber-physical platforms that must orchestrate complex software and hardware functions. The desired tasks for these systems include visual localization, depth estimation, 3D mapping, disease detection and segmentation, automated navigation, active control, path realization and optional therapeutic modules such as targeted drug delivery and biopsy sampling. Data-driven algorithms promise to enable many advanced functionalities for capsule endoscopes, but real-world data is challenging to obtain. Physically-realistic simulations providing synthetic data have emerged as a solution to the development of data-driven algorithms. In this work, we present a comprehensive simulation platform for capsule endoscopy operations and introduce VR-Caps, a virtual active capsule environment that simulates a range of normal and abnormal tissue conditions (e.g., inflated, dry, wet etc.) and varied organ types, capsule endoscope designs (e.g., mono, stereo, dual and 360degcamera), and the type, number, strength, and placement of internal and external magnetic sources that enable active locomotion. VR-Caps makes it possible to both independently or jointly develop, optimize, and test medical imaging and analysis software for the current and next-generation endoscopic capsule systems. To validate this approach, we train state-of-the-art deep neural networks to accomplish various medical image analysis tasks using simulated data from VR-Caps and evaluate the performance of these models on real medical data. Results demonstrate the usefulness and effectiveness of the proposed virtual platform in developing algorithms that quantify fractional coverage, camera trajectory, 3D map reconstruction, and disease classification.