# SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular images

arXiv: https://arxiv.org/abs/2101.07422 · HTML: https://mlsanity.com/paper/2101.07422/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 67 references that are in the map.
- First posted on arXiv 19 Jan 2021.

## Builds on (most cited first, 30 of 45)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Conditional Random Fields as Recurrent Neural Networks](https://mlsanity.com/paper/1502.03240/) (2015)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)

## Built on by (most cited first, 30 of 33)

- [DepthFM: Fast Monocular Depth Estimation with Flow Matching](https://mlsanity.com/paper/2403.13788/) (2024)
- [BridgeNet: A Joint Learning Network of Depth Map Super-Resolution and Monocular Depth Estimation](https://mlsanity.com/paper/2107.12541/) (2021)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [EVEN: An Event-Based Framework for Monocular Depth Estimation at Adverse Night Conditions](https://mlsanity.com/paper/2302.03860/) (2023)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [GCNDepth: Self-supervised Monocular Depth Estimation based on Graph Convolutional Network](https://mlsanity.com/paper/2112.06782/) (2021)
- [Few-Shot Learning by Integrating Spatial and Frequency Representation](https://mlsanity.com/paper/2105.05348/) (2021)
- [An Unsupervised Domain Adaptation Model based on Dual-module Adversarial Training](https://mlsanity.com/paper/2112.15555/) (2021)
- [Aggregating Global Features into Local Vision Transformer](https://mlsanity.com/paper/2201.12903/) (2022)
- [A Domain Gap Aware Generative Adversarial Network for Multi-domain Image Translation](https://mlsanity.com/paper/2110.10837/) (2021)
- [Joint Depth Prediction and Semantic Segmentation with Multi-View SAM](https://mlsanity.com/paper/2311.00134/) (2023)
- [SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2111.09692/) (2021)
- [Dynamic Label Assignment for Object Detection by Combining Predicted IoUs and Anchor IoUs](https://mlsanity.com/paper/2201.09396/) (2022)
- [On the Real-Time Semantic Segmentation of Aphid Clusters in the Wild](https://mlsanity.com/paper/2307.10267/) (2023)
- [Efficient Multi-task Uncertainties for Joint Semantic Segmentation and Monocular Depth Estimation](https://mlsanity.com/paper/2402.10580/) (2024)
- [DepthFake: a depth-based strategy for detecting Deepfake videos](https://mlsanity.com/paper/2208.11074/) (2022)
- [SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion](https://mlsanity.com/paper/2209.00381/) (2022)
- [A Discriminative Channel Diversification Network for Image Classification](https://mlsanity.com/paper/2112.05861/) (2021)
- [A New Dataset and Comparative Study for Aphid Cluster Detection and Segmentation in Sorghum Fields](https://mlsanity.com/paper/2405.04305/) (2024)
- [CI-Net: Contextual Information for Joint Semantic Segmentation and Depth Estimation](https://mlsanity.com/paper/2107.13800/) (2021)
- [Multiple Classifiers Based Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2108.00610/) (2021)
- [PanDepth: Joint Panoptic Segmentation and Depth Completion](https://mlsanity.com/paper/2212.14180/) (2022)
- [Kitchen Food Waste Image Segmentation and Classification for Compost Nutrients Estimation](https://mlsanity.com/paper/2401.15175/) (2024)
- [Multi-Layer Dense Attention Decoder for Polyp Segmentation](https://mlsanity.com/paper/2403.18180/) (2024)
- [A Comparative Study on Multi-task Uncertainty Quantification in Semantic Segmentation and Monocular Depth Estimation](https://mlsanity.com/paper/2405.17097/) (2024)
- [Vision-Driven 2D Supervised Fine-Tuning Framework for Bird's Eye View Perception](https://mlsanity.com/paper/2409.05834/) (2024)
- [Simpler Does It: Generating Semantic Labels with Objectness Guidance](https://mlsanity.com/paper/2110.10335/) (2021)
- [TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency](https://mlsanity.com/paper/2210.05152/) (2022)
- [Hybrid Transformer Based Feature Fusion for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2211.11066/) (2022)
- [DenseFormer: Learning Dense Depth Map from Sparse Depth and Image via Conditional Diffusion Model](https://mlsanity.com/paper/2503.23993/) (2025)

## Abstract (arXiv)

Depth estimation and semantic segmentation play essential roles in scene understanding. The state-of-the-art methods employ multi-task learning to simultaneously learn models for these two tasks at the pixel-wise level. They usually focus on sharing the common features or stitching feature maps from the corresponding branches. However, these methods lack in-depth consideration on the correlation of the geometric cues and the scene parsing. In this paper, we first introduce the concept of semantic objectness to exploit the geometric relationship of these two tasks through an analysis of the imaging process, then propose a Semantic Object Segmentation and Depth Estimation Network (SOSD-Net) based on the objectness assumption. To the best of our knowledge, SOSD-Net is the first network that exploits the geometry constraint for simultaneous monocular depth estimation and semantic segmentation. In addition, considering the mutual implicit relationship between these two tasks, we exploit the iterative idea from the expectation-maximization algorithm to train the proposed network more effectively. Extensive experimental results on the Cityscapes and NYU v2 dataset are presented to demonstrate the superior performance of the proposed approach.