# Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos

arXiv: https://arxiv.org/abs/2403.17915 · HTML: https://mlsanity.com/paper/2403.17915/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 71 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction](https://mlsanity.com/paper/1807.08865/) (2018)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/1704.02157/) (2017)
- [PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo](https://mlsanity.com/paper/2207.11406/) (2022)
- [EndoMapper dataset of complete calibrated endoscopy procedures](https://mlsanity.com/paper/2204.14240/) (2022)
- [Normal Assisted Stereo Depth Estimation](https://mlsanity.com/paper/1911.10444/) (2019)
- [Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods](https://mlsanity.com/paper/1902.07766/) (2019)
- [Shape and Material Capture at Home](https://mlsanity.com/paper/2104.06397/) (2021)
- [IronDepth: Iterative Refinement of Single-View Depth using Surface Normal and its Uncertainty](https://mlsanity.com/paper/2210.03676/) (2022)
- [A Differential Volumetric Approach to Multi-View Photometric Stereo](https://mlsanity.com/paper/1811.01984/) (2018)

## Built on by (most cited first, 14 of 14)

- [C3VDv2 -- Colonoscopy 3D video dataset with enhanced realism](https://mlsanity.com/paper/2506.24074/) (2025)
- [Building Rome with Convex Optimization](https://mlsanity.com/paper/2502.04640/) (2025)
- [NFL-BA: Near-Field Light Bundle Adjustment for SLAM in Dynamic Lighting](https://mlsanity.com/paper/2412.13176/) (2024)
- [3D Densification for Multi-Map Monocular VSLAM in Endoscopy](https://mlsanity.com/paper/2503.14346/) (2025)
- [PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation](https://mlsanity.com/paper/2504.17067/) (2025)
- [EndoStreamDepth: Temporally Consistent Monocular Depth Estimation for Endoscopic Video Streams](https://mlsanity.com/paper/2512.18159/) (2025)
- [Learning to Efficiently Adapt Foundation Models for Self-Supervised Endoscopic 3D Scene Reconstruction from Any Cameras](https://mlsanity.com/paper/2503.15917/) (2025)
- [Endoscopic Depth Estimation Based on Deep Learning: A Survey](https://mlsanity.com/paper/2507.20881/) (2025)
- [MetaScope: Optics-Driven Neural Network for Ultra-Micro Metalens Endoscopy](https://mlsanity.com/paper/2508.03596/) (2025)
- [EndoGMDE: Generalizable Monocular Depth Estimation with Mixture of Low-Rank Experts for Diverse Endoscopic Scenes](https://mlsanity.com/paper/2509.01206/) (2025)
- [EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model](https://mlsanity.com/paper/2510.22359/) (2025)
- [Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation](https://mlsanity.com/paper/2602.21740/) (2026)
- [Understanding Model Behavior in Monocular Polyp Sizing](https://mlsanity.com/paper/2605.20461/) (2026)
- [Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation](https://mlsanity.com/paper/2606.17340/) (2026)

## Abstract (arXiv)

Monocular depth estimation in endoscopy videos can enable assistive and robotic surgery to obtain better coverage of the organ and detection of various health issues. Despite promising progress on mainstream, natural image depth estimation, techniques perform poorly on endoscopy images due to a lack of strong geometric features and challenging illumination effects. In this paper, we utilize the photometric cues, i.e., the light emitted from an endoscope and reflected by the surface, to improve monocular depth estimation. We first create two novel loss functions with supervised and self-supervised variants that utilize a per-pixel shading representation. We then propose a novel depth refinement network (PPSNet) that leverages the same per-pixel shading representation. Finally, we introduce teacher-student transfer learning to produce better depth maps from both synthetic data with supervision and clinical data with self-supervision. We achieve state-of-the-art results on the C3VD dataset while estimating high-quality depth maps from clinical data. Our code, pre-trained models, and supplementary materials can be found on our project page: https://ppsnet.github.io/