Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction
Not on the map. Why?
Abstract (arXiv)
Vision foundation backbones provide strong representations for dense prediction, yet a single shared feature still needs to support tasks with different, image-dependent adaptation requirements. We propose MemMTL, a multi-task dense prediction framework that estimates a compact task state from global visual context and refines it through a learnable task-state prototype memory. The refined state is converted into task-conditioned expert logits and combined with token-level logits before sparse top-k selection over a local expert bank shared by all tasks. A separate task-agnostic residual bank provides a common adaptation path, and both paths are added once to the backbone feature before task-specific prediction. We specify a matched evaluation protocol on NYUD-v2 and PASCAL-Context with SAM 3 and ViT-L backbones to measure predictive quality, computational cost, and the contributions of task-state conditioning, prototype retrieval, and sparse routing. The numerical record in the present working draft predates this canonical implementation and must be regenerated before it can support empirical claims.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 28 Aug 2026.
Builds on
26 of 43 references.
References · 26
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
- 2021Scaling Vision with Sparse Mixture of Expertscited by 740
- 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
- 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
- 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
- 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 161
14 more
- 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
- 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
- 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
- 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
- 2021FaPN: Feature-aligned Pyramid Network for Dense Image Predictioncited by 81
- 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
- 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
- 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
- 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
- 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
- 2023DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Predictioncited by 27
- 2023Vision Transformer Adapters for Generalizable Multitask Learningcited by 14
- 2023Prompt Guided Transformer for Multi-Task Dense Predictioncited by 13
- 2025NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspectivecited by 2
Built on by
Not cited yet.