mlsanityData to 7 Oct 2026

Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners

arXiv 2212.08066 ↗First posted 15 Dec 2022cs.CV cs.AI cs.LGcited by 15

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Optimization in multi-task learning (MTL) is more challenging than single-task learning (STL), as the gradient from different tasks can be contradictory. When tasks are related, it can be beneficial to share some parameters among them (cooperation). However, some tasks require additional parameters with expertise in a specific type of data or discrimination (specialization). To address the MTL challenge, we propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad'). This structure allows us to formalize cooperation and specialization as the process of matching experts and tasks. We optimize this matching process during the training of a single model. Specifically, we incorporate mixture of experts (MoE) layers into a transformer model, with a new loss that incorporates the mutual dependence between tasks and experts. As a result, only a small set of experts are activated for each task. This prevents the sharing of the entire backbone model between all tasks, which strengthens the model, especially when the training set size and the number of tasks scale up. More interestingly, for each task, we can extract the small set of experts as a standalone model that maintains the same performance as the large model. Extensive experiments on the Taskonomy dataset with 13 vision tasks and the PASCAL-Context dataset with 5 vision tasks show the superiority of our approach.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 34 of its 40 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

34 of 40 references.

References · 34

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  8. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  10. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  12. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
22 more
  1. 2018Rethinking the Value of Network Pruningcited by 595
  2. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  3. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  4. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  5. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  6. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  7. 2018Slimmable Neural Networkscited by 301
  8. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  9. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  10. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  11. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  12. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  13. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  14. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  15. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  16. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  17. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  18. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  19. 2019Learning Filter Basis for Convolutional Neural Network Compressioncited by 57
  20. 2019Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernelscited by 42
  21. 2022Residual Mixture of Expertscited by 39
  22. 2019Deep Elastic Networks with Model Selection for Multi-Task Learningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 6; 2025: 0; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  2. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  3. 2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
  4. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  5. 2023Sparse Universal Transformercited by 28
  6. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  7. 2024DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Datacited by 18
  8. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  9. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  10. 2023Information Flow Control in Machine Learning through Modular Model Architecturecited by 11
  11. 2024Diffusion Models For Multi-Modal Generative Modelingcited by 9
  12. 2023Task-Based MoE for Multitask Multilingual Machine Translationcited by 5
3 more
  1. 2024Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddingscited by 1
  2. 2026RobuMTL: Enhancing Multi-Task Learning Robustness Against Weather Conditionscited by 1
  3. 2026Learngene Search Across Multiple Datasets for Building Variable-Sized Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
  2. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning
  3. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts
  4. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
  5. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
  6. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners
  7. 2025Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Experts
  8. 2025Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer