# MTLSI-Net: A Linear Semantic Interaction Network for Parameter-Efficient Multi-Task Dense Prediction

arXiv: https://arxiv.org/abs/2604.01995 · HTML: https://mlsanity.com/paper/2604.01995/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models](https://mlsanity.com/paper/2410.10733/) (2024)
- [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://mlsanity.com/paper/2001.06902/) (2020)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [Linear attention is (maybe) all you need (to understand transformer optimization)](https://mlsanity.com/paper/2310.01082/) (2023)
- [InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understanding](https://mlsanity.com/paper/2203.07997/) (2022)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning](https://mlsanity.com/paper/2003.14058/) (2020)
- [TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts](https://mlsanity.com/paper/2307.15324/) (2023)
- [Multi-Task Learning with Multi-Query Transformer for Dense Prediction](https://mlsanity.com/paper/2205.14354/) (2022)
- [DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction](https://mlsanity.com/paper/2301.03461/) (2023)
- [InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding](https://mlsanity.com/paper/2306.04842/) (2023)
- [AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning](https://mlsanity.com/paper/2110.13076/) (2021)
- [Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2407.08130/) (2024)
- [T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates](https://mlsanity.com/paper/2507.07633/) (2025)
- [Spiking Variational Graph Representation Inference for Video Summarization](https://mlsanity.com/paper/2508.15389/) (2025)
- [Language-Guided Graph Representation Learning for Video Summarization](https://mlsanity.com/paper/2511.10953/) (2025)
- [Tree-Structured Data Clustering-Driven Neural Network for Intra Prediction in Video Coding](https://mlsanity.com/paper/2106.05481/) (2021)
- [MRT: Learning Compact Representations with Mixed RWKV-Transformer for Extreme Image Compression](https://mlsanity.com/paper/2511.06717/) (2025)

## Abstract (arXiv)

Multi-task dense prediction aims to perform multiple pixel-level tasks simultaneously. However, capturing global cross-task interactions remains non-trivial due to the quadratic complexity of standard self-attention on high-resolution features. To address this limitation, we propose a Multi-Task Linear Semantic Interaction Network (MTLSI-Net), which facilitates cross-task interaction through linear attention. Specifically, MTLSI-Net incorporates three key components: a Multi-Task Multi-scale Query Linear Fusion Block, which captures cross-task dependencies across multiple scales with linear complexity using a shared global context matrix; a Semantic Token Distiller that compresses redundant features into compact semantic tokens, distilling essential cross-task knowledge; and a Cross-Window Integrated attention Block that injects global semantics into local features via a dual-branch architecture, preserving both global consistency and spatial precision. These components collectively enable the network to capture comprehensive cross-task interactions at linear complexity with reduced parameters. Extensive experiments on NYUDv2 and PASCAL-Context demonstrate that MTLSI-Net achieves state-of-the-art performance, validating its effectiveness and efficiency in multi-task learning.