# Multitask Network for Joint Object Detection, Semantic Segmentation and Human Pose Estimation in Vehicle Occupancy Monitoring

arXiv: https://arxiv.org/abs/2205.01515 · HTML: https://mlsanity.com/paper/2205.01515/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 27 of 27)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [CSPNet: A New Backbone that can Enhance Learning Capability of CNN](https://mlsanity.com/paper/1911.11929/) (2019)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning](https://mlsanity.com/paper/2010.05627/) (2020)
- [MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving](https://mlsanity.com/paper/1612.07695/) (2016)
- [MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network](https://mlsanity.com/paper/1807.04067/) (2018)
- [Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose](https://mlsanity.com/paper/1811.12004/) (2018)
- [BlitzNet: A Real-Time Deep Network for Scene Understanding](https://mlsanity.com/paper/1708.02813/) (2017)
- [Light-Weight RefineNet for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1810.03272/) (2018)
- [SVIRO: Synthetic Vehicle Interior Rear Seat Occupancy Dataset and Benchmark](https://mlsanity.com/paper/2001.03483/) (2020)
- [MultiTask-CenterNet (MCN): Efficient and Diverse Multitask Learning using an Anchor Free Approach](https://mlsanity.com/paper/2108.05060/) (2021)
- [TICaM: A Time-of-flight In-car Cabin Monitoring Dataset](https://mlsanity.com/paper/2103.11719/) (2021)
- [DFKI Cabin Simulator: A Test Platform for Visual In-Cabin Monitoring Functions](https://mlsanity.com/paper/2002.03749/) (2020)
- [Autoencoder Based Inter-Vehicle Generalization for In-Cabin Occupant Classification](https://mlsanity.com/paper/2105.03164/) (2021)

## Built on by (most cited first, 4 of 4)

- [Multi-Task Consistency for Active Learning](https://mlsanity.com/paper/2306.12398/) (2023)
- [Light-Weight Vision Transformer with Parallel Local and Global Self-Attention](https://mlsanity.com/paper/2307.09120/) (2023)
- [Transformer-based Detection of Microorganisms on High-Resolution Petri Dish Images](https://mlsanity.com/paper/2308.09436/) (2023)
- [Classifier Ensemble for Efficient Uncertainty Calibration of Deep Neural Networks for Image Classification](https://mlsanity.com/paper/2501.10089/) (2025)

## Abstract (arXiv)

In order to ensure safe autonomous driving, precise information about the conditions in and around the vehicle must be available. Accordingly, the monitoring of occupants and objects inside the vehicle is crucial. In the state-of-the-art, single or multiple deep neural networks are used for either object recognition, semantic segmentation, or human pose estimation. In contrast, we propose our Multitask Detection, Segmentation and Pose Estimation Network (MDSP) -- the first multitask network solving all these three tasks jointly in the area of occupancy monitoring. Due to the shared architecture, memory and computing costs can be saved while achieving higher accuracy. Furthermore, our architecture allows a flexible combination of the three mentioned tasks during a simple end-to-end training. We perform comprehensive evaluations on the public datasets SVIRO and TiCaM in order to demonstrate the superior performance.