# Analyzing Modular CNN Architectures for Joint Depth Prediction and Semantic Segmentation

arXiv: https://arxiv.org/abs/1702.08009 · HTML: https://mlsanity.com/paper/1702.08009/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2017.

## Built on by (most cited first, 8 of 8)

- [SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular images](https://mlsanity.com/paper/2101.07422/) (2021)
- [Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervision](https://mlsanity.com/paper/2103.12209/) (2021)
- [From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction](https://mlsanity.com/paper/2203.08122/) (2022)
- [GOSS: Towards Generalized Open-set Semantic Segmentation](https://mlsanity.com/paper/2203.12116/) (2022)
- [Real-Time Monocular Human Depth Estimation and Segmentation on Embedded Systems](https://mlsanity.com/paper/2108.10506/) (2021)
- [MTLSegFormer: Multi-task Learning with Transformers for Semantic Segmentation in Precision Agriculture](https://mlsanity.com/paper/2305.02813/) (2023)
- [Attention on Classification for Fire Segmentation](https://mlsanity.com/paper/2111.03129/) (2021)
- [Predicting Depth from Semantic Segmentation using Game Engine Dataset](https://mlsanity.com/paper/2106.15257/) (2021)

## Abstract (arXiv)

This paper addresses the task of designing a modular neural network architecture that jointly solves different tasks. As an example we use the tasks of depth estimation and semantic segmentation given a single RGB image. The main focus of this work is to analyze the cross-modality influence between depth and semantic prediction maps on their joint refinement. While most previous works solely focus on measuring improvements in accuracy, we propose a way to quantify the cross-modality influence. We show that there is a relationship between final accuracy and cross-modality influence, although not a simple linear one. Hence a larger cross-modality influence does not necessarily translate into an improved accuracy. We find that a beneficial balance between the cross-modality influences can be achieved by network architecture and conjecture that this relationship can be utilized to understand different network design choices. Towards this end we propose a Convolutional Neural Network (CNN) architecture that fuses the state of the state-of-the-art results for depth estimation and semantic labeling. By balancing the cross-modality influences between depth and semantic prediction, we achieve improved results for both tasks using the NYU-Depth v2 benchmark.