# Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescue

arXiv: https://arxiv.org/abs/2112.08122 · HTML: https://mlsanity.com/paper/2112.08122/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 73 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry](https://mlsanity.com/paper/2003.01060/) (2020)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [Feature-metric Loss for Self-supervised Learning of Depth and Egomotion](https://mlsanity.com/paper/2007.10603/) (2020)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume](https://mlsanity.com/paper/2003.13951/) (2020)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks](https://mlsanity.com/paper/1605.02305/) (2016)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)

## Built on by (most cited first, 30 of 72)

- [Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery](https://mlsanity.com/paper/2401.06013/) (2024)
- [Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting](https://mlsanity.com/paper/2401.16416/) (2024)
- [EndoDAC: Efficient Adapting Foundation Model for Self-Supervised Depth Estimation from Any Endoscopic Camera](https://mlsanity.com/paper/2405.08672/) (2024)
- [EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting](https://mlsanity.com/paper/2403.15124/) (2024)
- [Scaling up self-supervised learning for improved surgical foundation models](https://mlsanity.com/paper/2501.09436/) (2025)
- [URCDC-Depth: Uncertainty Rectified Cross-Distillation with CutFlip for Monocular Depth Estimation](https://mlsanity.com/paper/2302.08149/) (2023)
- [Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video](https://mlsanity.com/paper/2504.03198/) (2025)
- [Frontiers in Intelligent Colonoscopy](https://mlsanity.com/paper/2410.17241/) (2024)
- [Depth Anything in Medical Images: A Comparative Study](https://mlsanity.com/paper/2401.16600/) (2024)
- [Transferring Relative Monocular Depth to Surgical Vision with Temporal Consistency](https://mlsanity.com/paper/2403.06683/) (2024)
- [EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels](https://mlsanity.com/paper/2409.05442/) (2024)
- [Advancing Dense Endoscopic Reconstruction with Gaussian Splatting-driven Surface Normal-aware Tracking and Mapping](https://mlsanity.com/paper/2501.19319/) (2025)
- [MonoPCC: Photometric-invariant Cycle Constraint for Monocular Depth Estimation of Endoscopic Images](https://mlsanity.com/paper/2404.16571/) (2024)
- [WS-SfMLearner: Self-supervised Monocular Depth and Ego-motion Estimation on Surgical Videos with Unknown Camera Parameters](https://mlsanity.com/paper/2308.11776/) (2023)
- [A geometry-aware deep network for depth estimation in monocular endoscopy](https://mlsanity.com/paper/2304.10241/) (2023)
- [DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model](https://mlsanity.com/paper/2408.17433/) (2024)
- [C3Fusion: Consistent Contrastive Colon Fusion, Towards Deep SLAM in Colonoscopy](https://mlsanity.com/paper/2206.01961/) (2022)
- [Multi-Scale Structural-aware Exposure Correction for Endoscopic Imaging](https://mlsanity.com/paper/2210.15033/) (2022)
- [DD-VNB: A Depth-based Dual-Loop Framework for Real-time Visually Navigated Bronchoscopy](https://mlsanity.com/paper/2403.01683/) (2024)
- [Endo-FASt3r: Endoscopic Foundation model Adaptation for Structure from motion](https://mlsanity.com/paper/2503.07204/) (2025)
- [Mono3R: Exploiting Monocular Cues for Geometric 3D Reconstruction](https://mlsanity.com/paper/2504.13419/) (2025)
- [The Drunkard's Odometry: Estimating Camera Motion in Deforming Scenes](https://mlsanity.com/paper/2306.16917/) (2023)
- [Task-guided Domain Gap Reduction for Monocular Depth Prediction in Endoscopy](https://mlsanity.com/paper/2310.01663/) (2023)
- [Surgical Depth Anything: Depth Estimation for Surgical Scenes using Foundation Models](https://mlsanity.com/paper/2410.07434/) (2024)
- [BodySLAM: A Generalized Monocular Visual SLAM Framework for Surgical Applications](https://mlsanity.com/paper/2408.03078/) (2024)
- [Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling](https://mlsanity.com/paper/2408.07266/) (2024)
- [Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy](https://mlsanity.com/paper/2409.07723/) (2024)
- [A Novel Hybrid Endoscopic Dataset for Evaluating Machine Learning-based Photometric Image Enhancement Models](https://mlsanity.com/paper/2207.02396/) (2022)
- [SHADeS: Self-supervised Monocular Depth Estimation Through Non-Lambertian Image Decomposition](https://mlsanity.com/paper/2502.12994/) (2025)
- [EndoDepthL: Lightweight Endoscopic Monocular Depth Estimation with CNN-Transformer](https://mlsanity.com/paper/2308.02716/) (2023)

## Abstract (arXiv)

Recently, self-supervised learning technology has been applied to calculate depth and ego-motion from monocular videos, achieving remarkable performance in autonomous driving scenarios. One widely adopted assumption of depth and ego-motion self-supervised learning is that the image brightness remains constant within nearby frames. Unfortunately, the endoscopic scene does not meet this assumption because there are severe brightness fluctuations induced by illumination variations, non-Lambertian reflections and interreflections during data collection, and these brightness fluctuations inevitably deteriorate the depth and ego-motion estimation accuracy. In this work, we introduce a novel concept referred to as appearance flow to address the brightness inconsistency problem. The appearance flow takes into consideration any variations in the brightness pattern and enables us to develop a generalized dynamic image constraint. Furthermore, we build a unified self-supervised framework to estimate monocular depth and ego-motion simultaneously in endoscopic scenes, which comprises a structure module, a motion module, an appearance module and a correspondence module, to accurately reconstruct the appearance and calibrate the image brightness. Extensive experiments are conducted on the SCARED dataset and EndoSLAM dataset, and the proposed unified framework exceeds other self-supervised approaches by a large margin. To validate our framework's generalization ability on different patients and cameras, we train our model on SCARED but test it on the SERV-CT and Hamlyn datasets without any fine-tuning, and the superior results reveal its strong generalization ability. Code will be available at: urlhttps://github.com/ShuweiShao/AF-SfMLearner.