# MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

arXiv: https://arxiv.org/abs/2001.06902 · HTML: https://mlsanity.com/paper/2001.06902/

## Key facts

- Cited by 157 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2020.

## Built on by (most cited first, 30 of 157)

- [Geometry Uncertainty Projection Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2107.13774/) (2021)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understanding](https://mlsanity.com/paper/2203.07997/) (2022)
- [Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2104.13613/) (2021)
- [Auto-Lambda: Disentangling Dynamic Task Relationships](https://mlsanity.com/paper/2202.03091/) (2022)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://mlsanity.com/paper/2104.02745/) (2021)
- [VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding](https://mlsanity.com/paper/2312.08733/) (2023)
- [Multi-Task Dense Prediction via Mixture of Low-Rank Experts](https://mlsanity.com/paper/2403.17749/) (2024)
- [Fine-grained Semantics-aware Representation Enhancement for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2108.08829/) (2021)
- [Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2206.06363/) (2022)
- [Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Tasks](https://mlsanity.com/paper/2210.03265/) (2022)
- [TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts](https://mlsanity.com/paper/2307.15324/) (2023)
- [MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning](https://mlsanity.com/paper/2403.20320/) (2024)
- [Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://mlsanity.com/paper/2111.14893/) (2021)
- [Multi-Task Learning with Multi-Query Transformer for Dense Prediction](https://mlsanity.com/paper/2205.14354/) (2022)
- [Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments](https://mlsanity.com/paper/2207.04526/) (2022)
- [Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2105.07830/) (2021)
- [Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation](https://mlsanity.com/paper/2109.01801/) (2021)
- [Instance-wise Occlusion and Depth Orders in Natural Scenes](https://mlsanity.com/paper/2111.14562/) (2021)
- [SA-BEV: Generating Semantic-Aware Bird's-Eye-View Feature for Multi-view 3D Object Detection](https://mlsanity.com/paper/2307.11477/) (2023)
- [MGNet: Monocular Geometric Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2206.13199/) (2022)
- [DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction](https://mlsanity.com/paper/2301.03461/) (2023)
- [Dual-Balancing for Multi-Task Learning](https://mlsanity.com/paper/2308.12029/) (2023)
- [DenseMTL: Cross-task Attention Mechanism for Dense Multi-task Learning](https://mlsanity.com/paper/2206.08927/) (2022)
- [Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts](https://mlsanity.com/paper/2305.18691/) (2023)
- [Semi-supervised Multi-task Learning for Semantics and Depth](https://mlsanity.com/paper/2110.07197/) (2021)
- [InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding](https://mlsanity.com/paper/2306.04842/) (2023)
- [MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders](https://mlsanity.com/paper/2407.02228/) (2024)
- [PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation](https://mlsanity.com/paper/2301.00954/) (2023)

## Abstract (arXiv)

In this paper, we argue about the importance of considering task interactions at multiple scales when distilling task information in a multi-task learning setup. In contrast to common belief, we show that tasks with high affinity at a certain scale are not guaranteed to retain this behaviour at other scales, and vice versa. We propose a novel architecture, namely MTI-Net, that builds upon this finding in three ways. First, it explicitly models task interactions at every scale via a multi-scale multi-modal distillation unit. Second, it propagates distilled task information from lower to higher scales via a feature propagation module. Third, it aggregates the refined task features from all scales via a feature aggregation unit to produce the final per-task predictions. Extensive experiments on two multi-task dense labeling datasets show that, unlike prior work, our multi-task model delivers on the full potential of multi-task learning, that is, smaller memory footprint, reduced number of calculations, and better performance w.r.t. single-task learning. The code is made publicly available: https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch.