# SemHint-MD: Learning from Noisy Semantic Labels for Self-Supervised Monocular Depth Estimation

arXiv: https://arxiv.org/abs/2303.18219 · HTML: https://mlsanity.com/paper/2303.18219/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](https://mlsanity.com/paper/2203.03884/) (2022)
- [Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance](https://mlsanity.com/paper/2007.06936/) (2020)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Self-Supervised Monocular Depth Hints](https://mlsanity.com/paper/1909.09051/) (2019)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)
- [The Edge of Depth: Explicit Constraints between Segmentation and Depth](https://mlsanity.com/paper/2004.00171/) (2020)
- [SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation](https://mlsanity.com/paper/1810.01849/) (2018)
- [Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2109.12484/) (2021)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networks](https://mlsanity.com/paper/1604.07480/) (2016)
- [StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation](https://mlsanity.com/paper/2108.08574/) (2021)
- [Geometry meets semantics for semi-supervised monocular depth estimation](https://mlsanity.com/paper/1810.04093/) (2018)
- [Self-Supervised Depth Estimation in Laparoscopic Image using 3D Geometric Consistency](https://mlsanity.com/paper/2208.08407/) (2022)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)

## Built on by (most cited first, 1 of 1)

- [Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling](https://mlsanity.com/paper/2408.07266/) (2024)

## Abstract (arXiv)

Without ground truth supervision, self-supervised depth estimation can be trapped in a local minimum due to the gradient-locality issue of the photometric loss. In this paper, we present a framework to enhance depth by leveraging semantic segmentation to guide the network to jump out of the local minimum. Prior works have proposed to share encoders between these two tasks or explicitly align them based on priors like the consistency between edges in the depth and segmentation maps. Yet, these methods usually require ground truth or high-quality pseudo labels, which may not be easily accessible in real-world applications. In contrast, we investigate self-supervised depth estimation along with a segmentation branch that is supervised with noisy labels provided by models pre-trained with limited data. We extend parameter sharing from the encoder to the decoder and study the influence of different numbers of shared decoder parameters on model performance. Also, we propose to use cross-task information to refine current depth and segmentation predictions to generate pseudo-depth and semantic labels for training. The advantages of the proposed method are demonstrated through extensive experiments on the KITTI benchmark and a downstream task for endoscopic tissue deformation tracking.