# Multi-task learning with cross-task consistency for improved depth estimation in colonoscopy

arXiv: https://arxiv.org/abs/2311.18664 · HTML: https://mlsanity.com/paper/2311.18664/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018)
- [P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior](https://mlsanity.com/paper/2204.02091/) (2022)
- [Robust Learning Through Cross-Task Consistency](https://mlsanity.com/paper/2006.04096/) (2020)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [Recurrent Neural Network for (Un-)supervised Learning of Monocular VideoVisual Odometry and Depth](https://mlsanity.com/paper/1904.07087/) (2019)
- [Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry](https://mlsanity.com/paper/2112.08177/) (2021)
- [Adaptive Surface Normal Constraint for Depth Estimation](https://mlsanity.com/paper/2103.15483/) (2021)
- [Deep Learning and Conditional Random Fields-based Depth Estimation and Topographical Reconstruction from Conventional Endoscopy](https://mlsanity.com/paper/1710.11216/) (2017)
- [ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery](https://mlsanity.com/paper/2112.08189/) (2021)
- [CT-based COVID-19 Triage: Deep Multitask Learning Improves Joint Identification and Severity Quantification](https://mlsanity.com/paper/2006.01441/) (2020)
- [A geometry-aware deep network for depth estimation in monocular endoscopy](https://mlsanity.com/paper/2304.10241/) (2023)
- [ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2111.10371/) (2021)

## Built on by (most cited first, 8 of 8)

- [C3VDv2 -- Colonoscopy 3D video dataset with enhanced realism](https://mlsanity.com/paper/2506.24074/) (2025)
- [M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perception](https://mlsanity.com/paper/2510.17363/) (2025)
- [Self-supervised Monocular Depth and Pose Estimation for Endoscopy with Latent Priors](https://mlsanity.com/paper/2411.17790/) (2024)
- [Online camera-pose-free stereo endoscopic tissue deformation recovery with tissue-invariant vision-biomechanics consistency](https://mlsanity.com/paper/2506.19388/) (2025)
- [EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model](https://mlsanity.com/paper/2510.22359/) (2025)
- [MT-Depth: Multi-task Instance feature analysis for the Depth Completion](https://mlsanity.com/paper/2512.04734/) (2025)
- [Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation](https://mlsanity.com/paper/2602.21740/) (2026)
- [Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation](https://mlsanity.com/paper/2603.15374/) (2026)

## Abstract (arXiv)

Colonoscopy screening is the gold standard procedure for assessing abnormalities in the colon and rectum, such as ulcers and cancerous polyps. Measuring the abnormal mucosal area and its 3D reconstruction can help quantify the surveyed area and objectively evaluate disease burden. However, due to the complex topology of these organs and variable physical conditions, for example, lighting, large homogeneous texture, and image modality estimating distance from the camera aka depth) is highly challenging. Moreover, most colonoscopic video acquisition is monocular, making the depth estimation a non-trivial problem. While methods in computer vision for depth estimation have been proposed and advanced on natural scene datasets, the efficacy of these techniques has not been widely quantified on colonoscopy datasets. As the colonic mucosa has several low-texture regions that are not well pronounced, learning representations from an auxiliary task can improve salient feature extraction, allowing estimation of accurate camera depths. In this work, we propose to develop a novel multi-task learning (MTL) approach with a shared encoder and two decoders, namely a surface normal decoder and a depth estimator decoder. Our depth estimator incorporates attention mechanisms to enhance global context awareness. We leverage the surface normal prediction to improve geometric feature extraction. Also, we apply a cross-task consistency loss among the two geometrically related tasks, surface normal and camera depth. We demonstrate an improvement of 14.17% on relative error and 10.4% improvement on δ1 accuracy over the most accurate baseline state-of-the-art BTS approach. All experiments are conducted on a recently released C3VD dataset; thus, we provide a first benchmark of state-of-the-art methods.