# Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networks

arXiv: https://arxiv.org/abs/1604.07480 · HTML: https://mlsanity.com/paper/1604.07480/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2016.

## Built on by (most cited first, 30 of 36)

- [SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular images](https://mlsanity.com/paper/2101.07422/) (2021)
- [Deep Learning Based 3D Segmentation: A Survey](https://mlsanity.com/paper/2103.05423/) (2021)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervision](https://mlsanity.com/paper/2103.12209/) (2021)
- [From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction](https://mlsanity.com/paper/2203.08122/) (2022)
- [DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium](https://mlsanity.com/paper/2304.03560/) (2023)
- [PanopticDepth: A Unified Framework for Depth-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.00468/) (2022)
- [SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images](https://mlsanity.com/paper/2403.10662/) (2024)
- [RGB-D Salient Object Detection with Ubiquitous Target Awareness](https://mlsanity.com/paper/2109.03425/) (2021)
- [Joint Self-supervised Depth and Optical Flow Estimation towards Dynamic Objects](https://mlsanity.com/paper/2310.00011/) (2023)
- [Self-supervised Depth Estimation Leveraging Global Perception and Geometric Smoothness Using On-board Videos](https://mlsanity.com/paper/2106.03505/) (2021)
- [MonoDVPS: A Self-Supervised Monocular Depth Estimation Approach to Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2210.07577/) (2022)
- [Efficient Multi-task Uncertainties for Joint Semantic Segmentation and Monocular Depth Estimation](https://mlsanity.com/paper/2402.10580/) (2024)
- [Layered Depth Refinement with Mask Guidance](https://mlsanity.com/paper/2206.03048/) (2022)
- [Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds](https://mlsanity.com/paper/2109.02763/) (2021)
- [Hybridnet for depth estimation and semantic segmentation](https://mlsanity.com/paper/2402.06539/) (2024)
- [Image Vectorization with Depth: convexified shape layers with depth ordering](https://mlsanity.com/paper/2409.06648/) (2024)
- [Cross-Task Consistency Learning Framework for Multi-Task Learning](https://mlsanity.com/paper/2111.14122/) (2021)
- [Dilated Fully Convolutional Neural Network for Depth Estimation from a Single Image](https://mlsanity.com/paper/2103.07570/) (2021)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [A Dynamic Feature Interaction Framework for Multi-task Visual Perception](https://mlsanity.com/paper/2306.05061/) (2023)
- [A Comparative Study on Multi-task Uncertainty Quantification in Semantic Segmentation and Monocular Depth Estimation](https://mlsanity.com/paper/2405.17097/) (2024)
- [TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency](https://mlsanity.com/paper/2210.05152/) (2022)
- [SemHint-MD: Learning from Noisy Semantic Labels for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2303.18219/) (2023)
- [Depth-guided Free-space Segmentation for a Mobile Robot](https://mlsanity.com/paper/2311.01966/) (2023)
- [Deep Learning-Based 3D Instance and Semantic Segmentation: A Review](https://mlsanity.com/paper/2406.13308/) (2024)
- [Is there anything left? Measuring semantic residuals of objects removed from 3D Gaussian Splatting](https://mlsanity.com/paper/2503.17574/) (2025)
- [Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning](https://mlsanity.com/paper/2505.21231/) (2025)
- [Domain Adaptive Monocular Depth Estimation With Semantic Information](https://mlsanity.com/paper/2104.05764/) (2021)
- [Sparse Depth Completion with Semantic Mesh Deformation Optimization](https://mlsanity.com/paper/2112.05498/) (2021)

## Abstract (arXiv)

Multi-scale deep CNNs have been used successfully for problems mapping each pixel to a label, such as depth estimation and semantic segmentation. It has also been shown that such architectures are reusable and can be used for multiple tasks. These networks are typically trained independently for each task by varying the output layer(s) and training objective. In this work we present a new model for simultaneous depth estimation and semantic segmentation from a single RGB image. Our approach demonstrates the feasibility of training parts of the model for each task and then fine tuning the full, combined model on both tasks simultaneously using a single loss function. Furthermore we couple the deep CNN with fully connected CRF, which captures the contextual relationships and interactions between the semantic and depth cues improving the accuracy of the final results. The proposed model is trained and evaluated on NYUDepth V2 dataset outperforming the state of the art methods on semantic segmentation and achieving comparable results on the task of depth estimation.