mlsanityData to 7 Oct 2026

A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models

arXiv 2502.15828 ↗First posted 20 Feb 2025cs.LG cs.AIcited by 9

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

In order to streamline the fine-tuning of foundation models, Low-Rank Adapters (LoRAs) have been substantially adopted across various fields, including instruction tuning and domain adaptation. The underlying concept of LoRA involves decomposing a full-rank matrix into the product of two lower-rank matrices, which reduces storage consumption and accelerates the training process. Furthermore, to address the limited expressive capacity of LoRA, the Mixture-of-Expert (MoE) has been introduced for incorporating multiple LoRA adapters. The integration of LoRA experts leads to a visible improvement across several downstream scenes. However, the mixture of LoRAs (MoE-LoRA) still exhibits its low robustness during tuning and inferring. Inspired by the Riemannian Preconditioners which train LoRA as a sub-space projector, we propose a new training strategy for MoE-LoRA, to stabilize and boost its feature learning procedure by multi-space projections. Examinations on SGD and AdamW optimizers demonstrate the effectiveness of our methodology. Source code is available at https://github.com/THUDM/MoELoRA_Riemannian.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 39 of its 51 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

39 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts22 Apr
  2. 2024Higher Layers Need More LoRA Experts13 Feb
  3. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  4. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec
  5. 2023SiRA: Sparse Mixture of Low Rank Adaptation15 Nov

Other references · 34

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  8. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  9. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  10. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  11. 2024Mixture of LoRA Expertscited by 179
  12. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
22 more
  1. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  2. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  3. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
  4. 2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
  5. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  6. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  7. 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
  8. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  9. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  10. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  11. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  12. 2021Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurementscited by 28
  13. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 28
  14. 2020Low-Rank Matrix Recovery with Scaled Subgradient Methods: Fast and Robust Convergence Without the Condition Numbercited by 27
  15. 2022Preconditioned Gradient Descent for Overparameterized Nonconvex Burer--Monteiro Factorization with Global Optimality Certificationcited by 21
  16. 2014Riemannian preconditioningcited by 19
  17. 2024ResLoRA: Identity Residual Mapping in Low-Rank Adaptioncited by 18
  18. 2024LoRA Meets Dropout under a Unified Frameworkcited by 17
  19. 2024BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained Modelscited by 7
  20. 2023Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterizationcited by 6
  21. 2023Fast and Accurate Estimation of Low-Rank Matrices from Noisy Measurements via Preconditioned Non-Convex Gradient Descentcited by 5
  22. 2024SIBO: A Simple Booster for Parameter-Efficient Fine-Tuningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Revisiting semi-supervised learning in the era of foundation modelscited by 6
  2. 2026PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learningcited by 4
  3. 2026W2T: LoRA Weights Already Know What They Can Docited by 4
  4. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  5. 2025C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learningcited by 1
  6. 2026Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoningcited by 1
  7. 2025InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generationcited by 0
  8. 2026FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Modelscited by 0
  9. 2026RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  2. 2025LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
  3. 2025Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
  4. 2024MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
  5. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
  6. 2025LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation
  7. 2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning
  8. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment