# DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data

arXiv: https://arxiv.org/abs/2403.15389 · HTML: https://mlsanity.com/paper/2403.15389/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 47 of its 55 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout](https://mlsanity.com/paper/2010.06808/) (2020)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://mlsanity.com/paper/2001.06902/) (2020)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [Robust Learning Through Cross-Task Consistency](https://mlsanity.com/paper/2006.04096/) (2020)
- [CARD: Classification and Regression Diffusion Models](https://mlsanity.com/paper/2206.07275/) (2022)

## Built on by (most cited first, 18 of 18)

- [V2X-R: Cooperative LiDAR-4D Radar Fusion with Denoising Diffusion for 3D Object Detection](https://mlsanity.com/paper/2411.08402/) (2024)
- [GDSG: Graph Diffusion-based Solution Generator for Optimization Problems in MEC Networks](https://mlsanity.com/paper/2412.08296/) (2024)
- [Synchronizing Task Behavior: Aligning Multiple Tasks during Test-Time Training](https://mlsanity.com/paper/2507.07778/) (2025)
- [Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions](https://mlsanity.com/paper/2411.18823/) (2024)
- [Upcycling Text-to-Image Diffusion Models for Multi-Task Capabilities](https://mlsanity.com/paper/2503.11905/) (2025)
- [StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets](https://mlsanity.com/paper/2506.08013/) (2025)
- [Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Experts](https://mlsanity.com/paper/2507.19077/) (2025)
- [AortaDiff: A Unified Multitask Diffusion Framework For Contrast-Free AAA Imaging](https://mlsanity.com/paper/2510.01498/) (2025)
- [M2Diff: Multi-Modality Multi-Task Enhanced Diffusion Model for MRI-Guided Low-Dose PET Enhancement](https://mlsanity.com/paper/2603.09075/) (2026)
- [Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning](https://mlsanity.com/paper/2603.21111/) (2026)
- [SARe: Structure-Aware Generative 3D Fragment Reassembly](https://mlsanity.com/paper/2603.21611/) (2026)
- [Parameter Aware Mamba Model for Multi-task Dense Prediction](https://mlsanity.com/paper/2511.14503/) (2025)
- [NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networks](https://mlsanity.com/paper/2512.06251/) (2025)
- [Task Prototype-Based Knowledge Retrieval for Multi-Task Learning from Partially Annotated Data](https://mlsanity.com/paper/2601.07474/) (2026)
- [Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2605.19698/) (2026)
- [UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception](https://mlsanity.com/paper/2606.30332/) (2026)
- [MUSE: Unlocking Timestep as Native Task Steering for One-Step Dense Prediction](https://mlsanity.com/paper/2606.30370/) (2026)
- [Unified Video Dense Prediction from Disjoint Data](https://mlsanity.com/paper/2607.21592/) (2026)

## Abstract (arXiv)

Recently, there has been an increased interest in the practical problem of learning multiple dense scene understanding tasks from partially annotated data, where each training sample is only labeled for a subset of the tasks. The missing of task labels in training leads to low-quality and noisy predictions, as can be observed from state-of-the-art methods. To tackle this issue, we reformulate the partially-labeled multi-task dense prediction as a pixel-level denoising problem, and propose a novel multi-task denoising diffusion framework coined as DiffusionMTL. It designs a joint diffusion and denoising paradigm to model a potential noisy distribution in the task prediction or feature maps and generate rectified outputs for different tasks. To exploit multi-task consistency in denoising, we further introduce a Multi-Task Conditioning strategy, which can implicitly utilize the complementary nature of the tasks to help learn the unlabeled tasks, leading to an improvement in the denoising performance of the different tasks. Extensive quantitative and qualitative experiments demonstrate that the proposed multi-task denoising diffusion model can significantly improve multi-task prediction maps, and outperform the state-of-the-art methods on three challenging multi-task benchmarks, under two different partial-labeling evaluation settings. The code is available at https://prismformore.github.io/diffusionmtl/.