mlsanityData to 7 Oct 2026

Mixture of LoRA experts

128 papers0 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: lora, experts, fine-tuning, expert, continual

Key facts. Mixture of LoRA experts: 128 papers on arXiv since 2017 (data through 7 Oct 2026). 12 of its 128 mapped papers (9%) cite “Higher Layers Need More LoRA Experts” (2024, arXiv 2402.08562). 12 of its 128 mapped papers (9%) cite “Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters” (2024, arXiv 2403.11549). 3 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

3 new since 17 Sep

  1. Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation also fits: Multimodal MoE models
  2. S3: Spectral Null-Space Swap Makes Reasoning Models Efficient also fits: MoE large language models
  3. From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs also fits: MoE large language models

Foundations

Most cited by papers here

  1. Higher Layers Need More LoRA Experts2024cited by 12 here
  2. Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters2024cited by 12 here
  3. MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts2024cited by 12 here
  4. LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin2023cited by 9 here
  5. Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning2023cited by 5 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 41 here
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 18 here
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 9 here
  4. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 5 here
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 11 here
  6. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 9 here
  7. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 12 here
  8. 2024A Survey on Mixture of Experts in Large Language Modelscited by 5 here
16 more it builds on, not in this chain
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 34 here
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 8 here
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 10 here
  4. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 5 here
  5. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 5 here
  6. 2023From Sparse to Soft Mixtures of Expertscited by 5 here
  7. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 9 here
  8. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 5 here
  9. 2024Mixtral of Expertscited by 8 here
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 12 here
  11. 2024Higher Layers Need More LoRA Expertscited by 12 here
  12. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 7 here
  13. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 12 here
  14. 2024Theory on Mixture-of-Experts in Continual Learningcited by 5 here
  15. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 5 here
  16. 2024DeepSeek-V3 Technical Reportcited by 5 here

The 24 most cited of 148.

Browse

30 of 128, ranked 6–35

  1. 6Theory on Mixture-of-Experts in Continual Learning20245
  2. 7MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models20244
  3. 8Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models20244
  4. 9Mixture of Experts Meets Prompt-Based Continual Learning20244
  5. 10LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts20253
  6. 11SiRA: Sparse Mixture of Low Rank Adaptation20233
  7. 12SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning20262
  8. 13OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning20252
  9. 14A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models20252
  10. 15Enhancing Multimodal Continual Instruction Tuning with BranchLoRA20252
All 30
  1. 16LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs20242
  2. 17Multi-Task Dense Prediction via Mixture of Low-Rank Experts20242
  3. 18AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality20242
  4. 19MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning20242
  5. 20DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models20261
  6. 21Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning20261
  7. 22MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning20261
  8. 23RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts20251
  9. 24Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment20251
  10. 25LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models20251
  11. 26DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism20251
  12. 27A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning20251
  13. 28Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts20251
  14. 29Self-Evolving LLMs via Continual Instruction Tuning20251
  15. 30Merging Multi-Task Models via Weight-Ensembling Mixture of Experts20241
  16. 31X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design20241
  17. 32PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning20241
  18. 33Learning More Generalized Experts by Merging Experts in Mixture-of-Experts20241
  19. 34MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models20241
  20. 35TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts20231