# Efficient Multi-task Uncertainties for Joint Semantic Segmentation and Monocular Depth Estimation

arXiv: https://arxiv.org/abs/2402.10580 · HTML: https://mlsanity.com/paper/2402.10580/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 76 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 30 of 41)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://mlsanity.com/paper/2001.06902/) (2020)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [A General Framework for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/1907.06890/) (2019)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [Towards Real-Time Monocular Depth Estimation for Robotics: A Survey](https://mlsanity.com/paper/2111.08600/) (2021)
- [Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations](https://mlsanity.com/paper/1809.04766/) (2018)

## Built on by (most cited first, 6 of 6)

- [UNIC: Universal Classification Models via Multi-teacher Distillation](https://mlsanity.com/paper/2408.05088/) (2024)
- [EvidMTL: Evidential Multi-Task Learning for Uncertainty-Aware Semantic Surface Mapping from Monocular RGB Images](https://mlsanity.com/paper/2503.04441/) (2025)
- [Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness](https://mlsanity.com/paper/2506.05917/) (2025)
- [A Comparative Study on Multi-task Uncertainty Quantification in Semantic Segmentation and Monocular Depth Estimation](https://mlsanity.com/paper/2405.17097/) (2024)
- [Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks](https://mlsanity.com/paper/2501.12824/) (2025)
- [Optical aberrations in autonomous driving: Physics-informed parameterized temperature scaling for neural network uncertainty calibration](https://mlsanity.com/paper/2412.13695/) (2024)

## Abstract (arXiv)

Quantifying the predictive uncertainty emerged as a possible solution to common challenges like overconfidence or lack of explainability and robustness of deep neural networks, albeit one that is often computationally expensive. Many real-world applications are multi-modal in nature and hence benefit from multi-task learning. In autonomous driving, for example, the joint solution of semantic segmentation and monocular depth estimation has proven to be valuable. In this work, we first combine different uncertainty quantification methods with joint semantic segmentation and monocular depth estimation and evaluate how they perform in comparison to each other. Additionally, we reveal the benefits of multi-task learning with regard to the uncertainty quality compared to solving both tasks separately. Based on these insights, we introduce EMUFormer, a novel student-teacher distillation approach for joint semantic segmentation and monocular depth estimation as well as efficient multi-task uncertainty quantification. By implicitly leveraging the predictive uncertainties of the teacher, EMUFormer achieves new state-of-the-art results on Cityscapes and NYUv2 and additionally estimates high-quality predictive uncertainties for both tasks that are comparable or superior to a Deep Ensemble despite being an order of magnitude more efficient.