mlsanityData to 14 Oct 2026

Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Experts

arXiv 2507.19077 ↗First posted 25 Jul 2025cs.CVcited by 1

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Multi-task learning (MTL) for dense prediction has shown promising results but still faces challenges in balancing shared representations with task-specific specialization. In this paper, we introduce a novel Fine-Grained Mixture of Experts (FGMoE) architecture that explores MoE-based MTL models through a combination of three key innovations and fine-tuning. First, we propose intra-task experts that partition along intermediate hidden dimensions of MLPs, enabling finer decomposition of task information while maintaining parameter efficiency. Second, we introduce shared experts that consolidate common information across different contexts of the same task, reducing redundancy, and allowing routing experts to focus on unique aspects. Third, we design a global expert that facilitates adaptive knowledge transfer across tasks based on both input feature and task requirements, promoting beneficial information sharing while preventing harmful interference. In addition, we use the fine-tuning approach to improve parameter efficiency only by training the parameters of the decoder. Extensive experimental results show that the proposed FGMoE uses fewer parameters and significantly outperforms current MoE-based competitive MTL models on two dense prediction datasets (textiti.e., NYUD-v2, PASCAL-Context) in various metrics.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 32 of its 49 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

32 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Experts26 Mar
  2. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Experts1 Feb

Other references · 30

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  9. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  10. 2021Pay Attention to MLPscited by 368
  11. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  12. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
18 more
  1. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
  2. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  3. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  4. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  5. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 98
  6. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  7. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
  8. 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
  9. 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
  10. 2022Multi-Task Learning with Multi-Query Transformer for Dense Predictioncited by 34
  11. 2022Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Drivingcited by 30
  12. 2023DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Predictioncited by 27
  13. 2024DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Datacited by 18
  14. 2023Vision Transformer Adapters for Generalizable Multitask Learningcited by 14
  15. 2023Prompt Guided Transformer for Multi-Task Dense Predictioncited by 13
  16. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 9
  17. 2023Contrastive Multi-Task Dense Predictioncited by 8
  18. 2024Efficient Multitask Dense Predictor via Binarizationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hierarchical Wasserstein Merging for Multi-Domain Multi-Task Learning: From Specialists to a Generalistcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  2. 2024PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning
  3. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  4. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners
  5. 2024Scaling Laws for Fine-Grained Mixture of Experts
  6. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
  7. 2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning
  8. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts