# Learning Multiple Dense Prediction Tasks from Partially Annotated Data

arXiv: https://arxiv.org/abs/2111.14893 · HTML: https://mlsanity.com/paper/2111.14893/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 81 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.07936/) (2020)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)

## Built on by (most cited first, 30 of 42)

- [VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding](https://mlsanity.com/paper/2312.08733/) (2023)
- [TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts](https://mlsanity.com/paper/2307.15324/) (2023)
- [Multi-Task Learning with Multi-Query Transformer for Dense Prediction](https://mlsanity.com/paper/2205.14354/) (2022)
- [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://mlsanity.com/paper/2204.02744/) (2022)
- [InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding](https://mlsanity.com/paper/2306.04842/) (2023)
- [DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data](https://mlsanity.com/paper/2403.15389/) (2024)
- [PolyMaX: General Dense Prediction with Mask Transformer](https://mlsanity.com/paper/2311.05770/) (2023)
- [TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding](https://mlsanity.com/paper/2311.03427/) (2023)
- [StainDiffuser: MultiTask Dual Diffusion Model for Virtual Staining](https://mlsanity.com/paper/2403.11340/) (2024)
- [ProCC: Progressive Cross-primitive Compatibility for Open-World Compositional Zero-Shot Learning](https://mlsanity.com/paper/2211.12417/) (2022)
- [DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction](https://mlsanity.com/paper/2303.01573/) (2023)
- [Contrastive Multi-Task Dense Prediction](https://mlsanity.com/paper/2307.07934/) (2023)
- [Multi-task Learning with 3D-Aware Regularization](https://mlsanity.com/paper/2310.00986/) (2023)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [Joint-Task Regularization for Partially Labeled Multi-Task Learning](https://mlsanity.com/paper/2404.01976/) (2024)
- [Domain Generalisation for Object Detection under Covariate and Concept Shift](https://mlsanity.com/paper/2203.05294/) (2022)
- [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024)
- [Synchronizing Task Behavior: Aligning Multiple Tasks during Test-Time Training](https://mlsanity.com/paper/2507.07778/) (2025)
- [MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors](https://mlsanity.com/paper/2602.05330/) (2026)
- [T2M-X: Learning Expressive Text-to-Motion Generation from Partially Annotated Data](https://mlsanity.com/paper/2409.13251/) (2024)
- [Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions](https://mlsanity.com/paper/2411.18823/) (2024)
- [RepMode: Learning to Re-parameterize Diverse Experts for Subcellular Structure Prediction](https://mlsanity.com/paper/2212.10066/) (2022)
- [Data exploitation: multi-task learning of object detection and semantic segmentation on partially annotated data](https://mlsanity.com/paper/2311.04040/) (2023)
- [UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines](https://mlsanity.com/paper/2503.20748/) (2025)
- [SAMwave: Wavelet-Driven Feature Enrichment for Effective Adaptation of Segment Anything Model](https://mlsanity.com/paper/2507.20186/) (2025)
- [Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis](https://mlsanity.com/paper/2509.23915/) (2025)
- [Self-Training and Multi-Task Learning for Limited Data: Evaluation Study on Object Detection](https://mlsanity.com/paper/2309.06288/) (2023)
- [Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning](https://mlsanity.com/paper/2403.10252/) (2024)
- [Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks](https://mlsanity.com/paper/2501.12824/) (2025)
- [StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets](https://mlsanity.com/paper/2506.08013/) (2025)

## Abstract (arXiv)

Despite the recent advances in multi-task learning of dense prediction problems, most methods rely on expensive labelled datasets. In this paper, we present a label efficient approach and look at jointly learning of multiple dense prediction tasks on partially annotated data (i.e. not all the task labels are available for each image), which we call multi-task partially-supervised learning. We propose a multi-task training procedure that successfully leverages task relations to supervise its multi-task learning when data is partially annotated. In particular, we learn to map each task pair to a joint pairwise task-space which enables sharing information between them in a computationally efficient way through another network conditioned on task pairs, and avoids learning trivial cross-task relations by retaining high-level information about the input image. We rigorously demonstrate that our proposed method effectively exploits the images with unlabelled tasks and outperforms existing semi-supervised learning approaches and related methods on three standard benchmarks.