# DenseMTL: Cross-task Attention Mechanism for Dense Multi-task Learning

arXiv: https://arxiv.org/abs/2206.08927 · HTML: https://mlsanity.com/paper/2206.08927/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://mlsanity.com/paper/2001.06902/) (2020)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [DADA: Depth-aware Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1904.01886/) (2019)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2012.10782/) (2020)
- [Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency](https://mlsanity.com/paper/1711.03665/) (2017)
- [Distilled Semantics for Comprehensive Scene Understanding from Videos](https://mlsanity.com/paper/2003.14030/) (2020)
- [Geometric Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16694/) (2021)
- [Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2105.07830/) (2021)
- [Semi-supervised Multi-task Learning for Semantics and Depth](https://mlsanity.com/paper/2110.07197/) (2021)

## Built on by (most cited first, 25 of 25)

- [Material Palette: Extraction of Materials from a Single Image](https://mlsanity.com/paper/2311.17060/) (2023)
- [InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation](https://mlsanity.com/paper/2409.07914/) (2024)
- [Challenges for Monocular 6D Object Pose Estimation in Robotics](https://mlsanity.com/paper/2307.12172/) (2023)
- [Vision Transformer Adapters for Generalizable Multitask Learning](https://mlsanity.com/paper/2308.12372/) (2023)
- [SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images](https://mlsanity.com/paper/2403.10662/) (2024)
- [MICDrop: Masking Image and Depth Features via Complementary Dropout for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2408.16478/) (2024)
- [Cross-Task Attention Network: Improving Multi-Task Learning for Medical Imaging Applications](https://mlsanity.com/paper/2309.03837/) (2023)
- [M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perception](https://mlsanity.com/paper/2510.17363/) (2025)
- [Open Challenges for Monocular Single-shot 6D Object Pose Estimation](https://mlsanity.com/paper/2302.11827/) (2023)
- [X-PDNet: Accurate Joint Plane Instance Segmentation and Monocular Depth Estimation with Cross-Task Distillation and Boundary Correction](https://mlsanity.com/paper/2309.08424/) (2023)
- [Multi-Cohort Framework with Cohort-Aware Attention and Adversarial Mutual-Information Minimization for Whole Slide Image Classification](https://mlsanity.com/paper/2409.11119/) (2024)
- [Optimizing Dense Visual Predictions Through Multi-Task Coherence and Prioritization](https://mlsanity.com/paper/2412.03179/) (2024)
- [Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning](https://mlsanity.com/paper/2505.21231/) (2025)
- [LASS-ODE: Scaling ODE Computations to Connect Foundation Models with Dynamical Physical Systems](https://mlsanity.com/paper/2602.01009/) (2026)
- [MetaMorphosis: Task-oriented Privacy Cognizant Feature Generation for Multi-task Learning](https://mlsanity.com/paper/2305.07815/) (2023)
- [AutoTask: Task Aware Multi-Faceted Single Model for Multi-Task Ads Relevance](https://mlsanity.com/paper/2407.06549/) (2024)
- [MT-CYP-Net: Multi-Task Network for Pixel-Level Crop Yield Prediction Under Very Few Samples](https://mlsanity.com/paper/2505.12069/) (2025)
- [A Survey on Deep Multi-Task Learning in Connected Autonomous Vehicles](https://mlsanity.com/paper/2508.00917/) (2025)
- [Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Models](https://mlsanity.com/paper/2509.23626/) (2025)
- [Adaptive Bidirectional Task Interaction for Joint Segmentation and Classification of Breast Ultrasound](https://mlsanity.com/paper/2603.01295/) (2026)
- [Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance](https://mlsanity.com/paper/2603.07570/) (2026)
- [Semantic Aware Feature Extraction for Enhanced 3D Reconstruction](https://mlsanity.com/paper/2603.13556/) (2026)
- [PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts](https://mlsanity.com/paper/2605.14055/) (2026)
- [BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis](https://mlsanity.com/paper/2606.29102/) (2026)
- [DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction](https://mlsanity.com/paper/2607.16012/) (2026)

## Abstract (arXiv)

Multi-task learning has recently emerged as a promising solution for a comprehensive understanding of complex scenes. In addition to being memory-efficient, multi-task models, when appropriately designed, can facilitate the exchange of complementary signals across tasks. In this work, we jointly address 2D semantic segmentation and three geometry-related tasks: dense depth estimation, surface normal estimation, and edge estimation, demonstrating their benefits on both indoor and outdoor datasets. We propose a novel multi-task learning architecture that leverages pairwise cross-task exchange through correlation-guided attention and self-attention to enhance the overall representation learning for all tasks. We conduct extensive experiments across three multi-task setups, showing the advantages of our approach compared to competitive baselines in both synthetic and real-world benchmarks. Additionally, we extend our method to the novel multi-task unsupervised domain adaptation setting. Our code is available at https://github.com/cv-rits/DenseMTL