mlsanityData to 7 Oct 2026

Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints

arXiv 2103.16525 ↗First posted 30 Mar 2021cs.CV cs.LG cs.ROcited by 80

Not on the map. Why?

Abstract (arXiv)

Estimating a scene reconstruction and the camera motion from in-body videos is challenging due to several factors, e.g. the deformation of in-body cavities or the lack of texture. In this paper we present Endo-Depth-and-Motion, a pipeline that estimates the 6-degrees-of-freedom camera pose and dense 3D scene models from monocular endoscopic videos. Our approach leverages recent advances in self-supervised depth networks to generate pseudo-RGBD frames, then tracks the camera pose using photometric residuals and fuses the registered depth maps in a volumetric representation. We present an extensive experimental evaluation in the public dataset Hamlyn, showing high-quality results and comparisons against relevant baselines. We also release all models and code for future comparisons.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 30 of its 66 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

30 of 66 references.

References · 30

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  3. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  4. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  5. 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 801
  6. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  7. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  8. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  9. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  10. 2016Direct Sparse Odometrycited by 479
  11. 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 405
  12. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
18 more
  1. 2018Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstructioncited by 159
  2. 2017CNN-SLAM: Real-time dense monocular SLAM with learned depth predictioncited by 152
  3. 2020DeepFactors: Real-Time Probabilistic Dense Monocular SLAMcited by 117
  4. 2017Semi-Supervised Deep Learning for Monocular Depth Map Predictioncited by 109
  5. 2019CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depthcited by 75
  6. 2018Learning Monocular Depth by Distilling Cross-domain Stereo Networkscited by 64
  7. 2019To Learn or Not to Learn: Visual Localization from Essential Matricescited by 61
  8. 2017Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Trainingcited by 57
  9. 2019Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methodscited by 45
  10. 2019DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequencescited by 36
  11. 2018MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computingcited by 29
  12. 2019SLAM Endoscopy enhanced by adversarial depth predictioncited by 26
  13. 2020Dynamic Reconstruction of Deformable Soft-tissue with Stereo Scope in Minimal Invasive Surgerycited by 23
  14. 2020VR-Caps: A Virtual Environment for Capsule Endoscopycited by 23
  15. 2020SD-DefSLAM: Semi-Direct Monocular SLAM for Deformable and Intracorporeal Scenescited by 21
  16. 2016Single-View and Multi-View Depth Fusioncited by 10
  17. 2018Deep Learning with Cinematic Rendering: Fine-Tuning Deep Neural Networks Using Photorealistic Medical Imagescited by 10
  18. 2018Loosely-Coupled Semi-Direct Monocular SLAMcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 13; 2024: 23; 2025: 25; 2026: 9.

80 papers.

Sort:

By area:

  1. 2022Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgerycited by 122
  2. 2022Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registrationcited by 78
  3. 2021Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescuecited by 70
  4. 2024Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgerycited by 52
  5. 2024EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Cameracited by 42
  6. 2023Tracking and Mapping in Medical Computer Vision: A Reviewcited by 36
  7. 2024EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splattingcited by 33
  8. 2024EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splattingcited by 33
  9. 2023Efficient Deformable Tissue Reconstruction via Orthogonal Neural Planecited by 20
  10. 2022Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Trackingcited by 19
  11. 2023LightDepth: Single-View Depth Self-Supervision from Illumination Declinecited by 17
  12. 2023Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mappingcited by 15
38 more
  1. 2023SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Trackingcited by 14
  2. 2024Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videoscited by 14
  3. 2025Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Videocited by 14
  4. 2025C3VDv2 -- Colonoscopy 3D video dataset with enhanced realismcited by 14
  5. 2024Depth Anything in Medical Images: A Comparative Studycited by 13
  6. 2024Transferring Relative Monocular Depth to Surgical Vision with Temporal Consistencycited by 13
  7. 2024EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labelscited by 12
  8. 2021Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgerycited by 11
  9. 2022Photometric single-view dense 3D reconstruction in endoscopycited by 11
  10. 2025Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mappingcited by 11
  11. 2022SfM-TTR: Using Structure from Motion for Test-Time Refinement of Single-View Depth Networkscited by 10
  12. 2024Motion-Guided Dual-Camera Tracker for Endoscope Tracking and Motion Analysis in a Mechanical Gastric Simulatorcited by 10
  13. 2024MonoPCC: Photometric-invariant Cycle Constraint for Monocular Depth Estimation of Endoscopic Imagescited by 10
  14. 2025EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splattingcited by 10
  15. 2023A geometry-aware deep network for depth estimation in monocular endoscopycited by 8
  16. 2023Multi-task learning with cross-task consistency for improved depth estimation in colonoscopycited by 8
  17. 2024UC-NeRF: Uncertainty-aware Conditional Neural Radiance Fields from Endoscopic Sparse Viewscited by 8
  18. 2021On the Uncertain Single-View Depths in Colonoscopiescited by 6
  19. 2024PANS: Probabilistic Airway Navigation System for Real-time Robust Bronchoscope Localizationcited by 6
  20. 2025Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenescited by 6
  21. 2023The Drunkard's Odometry: Estimating Camera Motion in Deforming Scenescited by 5
  22. 2023BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fieldscited by 5
  23. 2025Endo-FASt3r: Endoscopic Foundation model Adaptation for Structure from motioncited by 5
  24. 2024BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applicationscited by 4
  25. 2024Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modelingcited by 4
  26. 2024Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopycited by 4
  27. 2025VSLAM-LAB: A Comprehensive Framework for Visual SLAM Methods and Datasetscited by 4
  28. 2025EndoWave: 4D Gaussian Splatting with Rational Wavelet for Endoscopic Reconstructioncited by 4
  29. 20223D endoscopic depth estimation using 3D surface-aware constraintscited by 3
  30. 2022A comprehensive survey on recent deep learning-based methods applied to surgical datacited by 3
  31. 2024Ada-Tracker: Soft Tissue Tracking via Inter-Frame and Adaptive-Template Matchingcited by 3
  32. 2024FLex: Joint Pose and Dynamic Radiance Fields Optimization for Stereo Endoscopic Videoscited by 3
  33. 20253D Densification for Multi-Map Monocular VSLAM in Endoscopycited by 3
  34. 2023SuPerPM: A Surgical Perception Framework Based on Deep Point Matching Learned from Physical Constrained Simulation Datacited by 2
  35. 2024Registering Neural 4D Gaussians for Endoscopic Surgerycited by 2
  36. 2024EndoDepth: A Benchmark for Assessing Robustness in Endoscopic Depth Predictioncited by 2
  37. 2025Zero-shot Monocular Metric Depth for Endoscopic Imagescited by 2
  38. 2025ColonAdapter: Geometry Estimation Through Foundation Model Adaptation for Colonoscopycited by 2

50 most cited of 80 All 80 (JSON)