mlsanityData to 7 Oct 2026

Heterogeneous Multi-task Learning with Expert Diversity

arXiv 2106.10595 ↗First posted 20 Jun 2021cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Predicting multiple heterogeneous biological and medical targets is a challenge for traditional deep learning models. In contrast to single-task learning, in which a separate model is trained for each target, multi-task learning (MTL) optimizes a single model to predict multiple related targets simultaneously. To address this challenge, we propose the Multi-gate Mixture-of-Experts with Exclusivity (MMoEEx). Our work aims to tackle the heterogeneous MTL setting, in which the same model optimizes multiple tasks with different characteristics. Such a scenario can overwhelm current MTL approaches due to the challenges in balancing shared and task-specific representations and the need to optimize tasks with competing optimization paths. Our method makes two key contributions: first, we introduce an approach to induce more diversity among experts, thus creating representations more suitable for highly imbalanced and heterogenous MTL learning; second, we adopt a two-step optimization [6, 11] approach to balancing the tasks at the gradient level. We validate our method on three MTL benchmark datasets, including Medical Information Mart for Intensive Care (MIMIC-III) and PubChem BioAssay (PCBA).

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 14 of its 25 references that are in the map. First posted on arXiv 20 Jun 2021.

Builds on

14 of 25 references.

References · 14

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  5. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  6. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  7. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  8. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  9. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  10. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  11. 2019Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and Morecited by 29
  12. 2020Multi-Task Learning for Dense Prediction Tasks: A Surveycited by 28
2 more
  1. 2021Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generationcited by 14
  2. 2020Multitask Learning with Single Gradient Step Update for Task Balancingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 4; 2024: 6; 2025: 5; 2026: 1.

18 papers.

Sort:

By area:

  1. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  2. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  3. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  4. 2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
  5. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  6. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  7. 2024Multimodal Clinical Trial Outcome Prediction with Large Language Modelscited by 7
  8. 2024FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Predictioncited by 6
  9. 2023DynaShare: Task and Instance Conditioned Parameter Sharing for Multi-Task Learningcited by 3
  10. 2023Exploiting Graph Structured Cross-Domain Representation for Multi-Domain Recommendationcited by 2
  11. 2024Context-Aware Token Selection and Packing for Enhanced Vision Transformercited by 1
  12. 2025The Transition from Centralized Machine Learning to Federated Learning for Mental Health in Education: A Survey of Current Methods and Future Directionscited by 1
6 more
  1. 2025Mixture of Group Experts for Learning Invariant Representationscited by 1
  2. 2025EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgerycited by 1
  3. 2025Robust-Multi-Task Gradient Boostingcited by 1
  4. 2021Multi-treatment Effect Estimation from Biomedical Datacited by 0
  5. 2025ColNet: Collaborative Optimization in Decentralized Federated Multi-task Learning Systemscited by 0
  6. 2026Bridging Adversarial and Collaborative Learning for AI-Generated Image Quality Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Experts
  2. 2024MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts
  3. 2022Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners
  4. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  5. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
  6. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
  7. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
  8. 2022Automatic Expert Selection for Multi-Scenario and Multi-Task Search