# Task Prototype-Based Knowledge Retrieval for Multi-Task Learning from Partially Annotated Data

arXiv: https://arxiv.org/abs/2601.07474 · HTML: https://mlsanity.com/paper/2601.07474/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Attentive Single-Tasking of Multiple Tasks](https://mlsanity.com/paper/1904.08918/) (2019)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)
- [NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction](https://mlsanity.com/paper/1801.08297/) (2018)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification](https://mlsanity.com/paper/1611.05377/) (2016)
- [Multi-Task Dense Prediction via Mixture of Low-Rank Experts](https://mlsanity.com/paper/2403.17749/) (2024)
- [TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts](https://mlsanity.com/paper/2307.15324/) (2023)
- [Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://mlsanity.com/paper/2111.14893/) (2021)
- [MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders](https://mlsanity.com/paper/2407.02228/) (2024)
- [DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data](https://mlsanity.com/paper/2403.15389/) (2024)
- [TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding](https://mlsanity.com/paper/2311.03427/) (2023)

## Abstract (arXiv)

Multi-task learning (MTL) is critical in real-world applications such as autonomous driving and robotics, enabling simultaneous handling of diverse tasks. However, obtaining fully annotated data for all tasks is impractical due to labeling costs. Existing methods for partially labeled MTL typically rely on predictions from unlabeled tasks, making it difficult to establish reliable task associations and potentially leading to negative transfer and suboptimal performance. To address these issues, we propose a prototype-based knowledge retrieval framework that achieves robust MTL instead of relying on predictions from unlabeled tasks. Our framework consists of two key components: (1) a task prototype embedding task-specific characteristics and quantifying task associations, and (2) a knowledge retrieval transformer that adaptively refines feature representations based on these associations. To achieve this, we introduce an association knowledge generating (AKG) loss to ensure the task prototype consistently captures task-specific characteristics. Extensive experiments demonstrate the effectiveness of our framework, highlighting its potential for robust multi-task learning, even when only a subset of tasks is annotated.