# SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion

arXiv: https://arxiv.org/abs/2209.00381 · HTML: https://mlsanity.com/paper/2209.00381/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 44 references that are in the map.
- First posted on arXiv 1 Sep 2022.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Gated-SCNN: Gated Shape CNNs for Semantic Segmentation](https://mlsanity.com/paper/1907.05740/) (2019)
- [Deep Parametric Continuous Convolutional Neural Networks](https://mlsanity.com/paper/2101.06742/) (2021)
- [Video Panoptic Segmentation](https://mlsanity.com/paper/2006.11339/) (2020)
- [Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera](https://mlsanity.com/paper/1807.00275/) (2018)
- [DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image](https://mlsanity.com/paper/1812.00488/) (2018)
- [Learning Guided Convolutional Network for Depth Completion](https://mlsanity.com/paper/1908.01238/) (2019)
- [EfficientPS: Efficient Panoptic Segmentation](https://mlsanity.com/paper/2004.02307/) (2020)
- [Deep Depth Completion of a Single RGB-D Image](https://mlsanity.com/paper/1803.09326/) (2018)
- [Auxiliary Tasks in Multi-task Learning](https://mlsanity.com/paper/1805.06334/) (2018)
- [Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints](https://mlsanity.com/paper/1910.06727/) (2019)
- [Searching for Efficient Multi-Scale Architectures for Dense Image Prediction](https://mlsanity.com/paper/1809.04184/) (2018)
- [Learning Joint 2D-3D Representations for Depth Completion](https://mlsanity.com/paper/2012.12402/) (2020)

## Built on by (most cited first, 5 of 5)

- [RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2309.00655/) (2023)
- [PanDepth: Joint Panoptic Segmentation and Depth Completion](https://mlsanity.com/paper/2212.14180/) (2022)
- [Efficient Semantic Splatting for Remote Sensing Multi-view Segmentation](https://mlsanity.com/paper/2412.05969/) (2024)
- [GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depth](https://mlsanity.com/paper/2604.11585/) (2026)
- [MT-Depth: Multi-task Instance feature analysis for the Depth Completion](https://mlsanity.com/paper/2512.04734/) (2025)

## Abstract (arXiv)

Holistic scene understanding is pivotal for the performance of autonomous machines. In this paper we propose a new end-to-end model for performing semantic segmentation and depth completion jointly. The vast majority of recent approaches have developed semantic segmentation and depth completion as independent tasks. Our approach relies on RGB and sparse depth as inputs to our model and produces a dense depth map and the corresponding semantic segmentation image. It consists of a feature extractor, a depth completion branch, a semantic segmentation branch and a joint branch which further processes semantic and depth information altogether. The experiments done on Virtual KITTI 2 dataset, demonstrate and provide further evidence, that combining both tasks, semantic segmentation and depth completion, in a multi-task network can effectively improve the performance of each task. Code is available at https://github.com/juanb09111/semantic depth.