mlsanityData to 14 Oct 2026

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

arXiv 2310.18339 ↗First posted 21 Oct 2023cs.CL cs.AIcited by 111

On the map: Mixture-of-experts → Medical and biomedical MoE

Abstract (arXiv)

The recent surge in Large Language Models (LLMs) has garnered significant attention across numerous fields. Fine-tuning is often required to fit general LLMs for a specific domain, like the web-based healthcare system. However, two problems arise during fine-tuning LLMs for medical applications. One is the task variety problem, which involves distinct tasks in real-world medical scenarios. The variety often leads to sub-optimal fine-tuning for data imbalance and seesaw problems. Besides, the large amount of parameters in LLMs leads to huge time and computation consumption by fine-tuning. To address these two problems, we propose a novel parameter efficient fine-tuning framework for multi-task medical applications, dubbed as MOELoRA. The designed framework aims to absorb both the benefits of mixture-of-expert (MOE) for multi-task learning and low-rank adaptation (LoRA) for parameter efficient fine-tuning. For unifying MOE and LoRA, we devise multiple experts as the trainable parameters, where each expert consists of a pair of low-rank matrices to retain the small size of trainable parameters. Then, a task-motivated gate function for all MOELoRA layers is proposed, which can control the contributions of each expert and produce distinct parameters for various tasks. We conduct experiments on a multi-task medical dataset, indicating MOELoRA outperforms the existing parameter efficient fine-tuning methods. The code is available online.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Placed in Medical and biomedical MoE, one of 12 directions in Mixture-of-experts. Builds on 33 of its 66 references that are in the map. First posted on arXiv 21 Oct 2023.

Builds on

33 of 66 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
21 more
  1. 2017A Survey on Multi-Task Learningcited by 840
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  3. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  4. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  7. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  8. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  9. 2019Dice Loss for Data-imbalanced NLP Taskscited by 182
  10. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 162
  11. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  12. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 141
  13. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  14. 2021One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Predictioncited by 78
  15. 2019Macro F1 and Macro F1cited by 70
  16. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  17. 2022A Comprehensive Survey on Trustworthy Recommender Systemscited by 32
  18. 2023Multi-Task Deep Recommender Systems: A Surveycited by 30
  19. 2023Multi-Task Recommendations with Reinforcement Learningcited by 26
  20. 2022Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfercited by 22
  21. 2018A Neural Multi-Task Learning Framework to Jointly Model Medical Named Entity Recognition and Normalizationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 32; 2025: 49; 2026: 27.

111 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  3. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  4. 2023Large Language Models for Generative Information Extraction: A Surveycited by 185
  5. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  6. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  7. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
  8. 2024Higher Layers Need More LoRA Expertscited by 102
  9. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  10. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  11. 2024A Survey on LoRA of Large Language Modelscited by 67
  12. 2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agentcited by 52
38 more
  1. 2024LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendationcited by 47
  2. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  3. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  4. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  5. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
  6. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  7. 2024Large Language Model Distilling Medication Recommendation Modelcited by 26
  8. 2026SkillGrad: Optimizing Agent Skills Like Gradient Descentcited by 26
  9. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  10. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  11. 2024LLM4MSR: An LLM-Enhanced Paradigm for Multi-Scenario Recommendationcited by 19
  12. 2024Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoEcited by 19
  13. 2023Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLMcited by 18
  14. 2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-designcited by 18
  15. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 18
  16. 2024MTL-LoRA: Low-Rank Adaptation for Multi-Task Learningcited by 17
  17. 2024Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarioscited by 17
  18. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  19. 2024Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Networkcited by 15
  20. 2025Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendationcited by 15
  21. 2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 13
  22. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  23. 2024Editing Factual Knowledge and Explanatory Ability of Medical Large Language Modelscited by 10
  24. 2025CoPL: Collaborative Preference Learning for Personalizing LLMscited by 10
  25. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  26. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  27. 2024MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Expertscited by 7
  28. 2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learningcited by 7
  29. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Modelscited by 7
  30. 2024Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graphcited by 6
  31. 2025Patient-Level Anatomy Meets Scanning-Level Physics: Personalized Federated Low-Dose CT Denoising Empowered by Large Language Modelcited by 6
  32. 2025Training-free LLM Merging for Multi-task Learningcited by 6
  33. 2024Mitigating Hallucinations of Large Language Models in Medical Information Extraction via Contrastive Decodingcited by 5
  34. 2024MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learningcited by 5
  35. 2025Gamma: Toward Generic Image Assessment with Mixture of Assessment Expertscited by 5
  36. 2025Guiding LLM-based Smart Contract Generation with Finite State Machinecited by 5
  37. 2025CoLA: Collaborative Low-Rank Adaptationcited by 5
  38. 2025TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preferencecited by 5

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptation
  2. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models
  3. 2024Can LLMs' Tuning Methods Work in Medical Multimodal Domain?
  4. 2024MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models
  5. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts
  6. 2024MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation
  7. 2025Generalization of Medical Large Language Models through Cross-Domain Weak Supervision
  8. 2023From Beginner to Expert: Modeling Medical Knowledge into General LLMs