mlsanityData to 7 Oct 2026

Mixture of LoRA Experts

arXiv 2404.13628 ↗First posted 21 Apr 2024cs.CL cs.LG cs.MMcited by 179

Not on the map. Why?

Cited by 38 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. Direct arithmetic merging may result in the loss of the original pre-trained model's generative capabilities or the distinct identity of LoRAs, thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based fusion exhibits limitations concerning the requisite flexibility for the effective combination of multiple LoRAs. In response to these challenges, this paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach not only achieves superior LoRA fusion performance in comparison to direct arithmetic merging but also retains the crucial flexibility for combining LoRAs effectively. Extensive experimental evaluations conducted in both the Natural Language Processing (NLP) and Vision & Language (V&L) domains substantiate the efficacy of MoLE.

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Builds on 21 of its 23 references that are in the map. First posted on arXiv 21 Apr 2024.

Builds on

21 of 23 references.

References · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
9 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  4. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  5. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  6. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  7. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  8. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  9. 2022MoEC: Mixture of Expert Clusterscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 81; 2026: 74.

179 papers.

Sort:

By area:

  1. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  2. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  3. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
  4. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  5. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answeringcited by 36
  6. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  7. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 32
  8. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  9. 2025SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Trackingcited by 23
  10. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  11. 2024Closed-form merging of parameter-efficient modules for Federated Continual Learningcited by 22
  12. 2025CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Mergingcited by 22
38 more
  1. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  2. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  3. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  4. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
  5. 2025Decouple and Orthogonalize: A Data-Free Framework for LoRA Mergingcited by 17
  6. 2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
  7. 2025What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?cited by 13
  8. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  9. 2024UniRestorer: Universal Image Restoration via Adaptively Estimating Image Degradation at Proper Granularitycited by 12
  10. 2025Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integrationcited by 12
  11. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 12
  12. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  13. 2025Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflictscited by 11
  14. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 11
  15. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
  16. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  17. 2025Ensembles of Low-Rank Expert Adapterscited by 9
  18. 2025EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inferencecited by 9
  19. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  20. 2025Ravan: Multi-Head Low-Rank Adaptation for Federated Fine-Tuningcited by 9
  21. 2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
  22. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  23. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  24. 2024MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shardscited by 7
  25. 2025Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDPcited by 7
  26. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  27. 2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialistscited by 6
  28. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  29. 2025Towards All-in-One Medical Image Re-Identificationcited by 6
  30. 2025MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidancecited by 6
  31. 2025Why Do More Experts Fail? A Theoretical Analysis of Model Mergingcited by 6
  32. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  33. 2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptationcited by 6
  34. 2026Effective LoRA Adapter Routing using Task Representationscited by 6
  35. 2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
  36. 2024LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Paircited by 5
  37. 2025Subject or Style: Adaptive and Training-Free Mixture of LoRAscited by 5
  38. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4

50 most cited of 179 All 179 (JSON)