mlsanityData to 7 Oct 2026

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

arXiv 2410.18035 ↗First posted 23 Oct 2024cs.CLcited by 19

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. However, they introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules in the Transformer layer. To address this issue, we propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism. This mechanism calculates expert routing results once before generating the first new token and reuses these results for subsequent tokens, reducing latency. Extensive experiments and analysis on commonsense reasoning tasks, math reasoning tasks, and widely used LLM evaluation benchmarks demonstrate that MiLoRA consistently outperforms strong PEFT baselines with comparable tunable parameter budgets. Additionally, MiLoRA significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 43 of its 85 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

43 of 85 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  2. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
29 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  10. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  11. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  12. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  13. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  14. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  15. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  16. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  17. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  18. 2019Pad'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networkscited by 65
  19. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  20. 2023PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domaincited by 40
  21. 2022Late Prompt Tuning: A Late Prompt Could Be Better Than Many Promptscited by 23
  22. 2023F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Taskscited by 18
  23. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 16
  24. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  25. 2023Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translationcited by 13
  26. 2020AutoTrans: Automating Transformer Design via Reinforced Architecture Searchcited by 12
  27. 2022Continuous Detection, Rapidly React: Unseen Rumors Detection based on Continual Prompt-Tuningcited by 12
  28. 2020AutoRC: Improving BERT Based Relation Classification Models via Architecture Searchcited by 9
  29. 2024PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdificationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 2.

19 papers.

Sort:

By area:

  1. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  2. 2025UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Modelscited by 11
  3. 2025StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenescited by 8
  4. 2025FT-MDT: Extracting Decision Trees from Medical Texts via a Novel Low-rank Adaptation Methodcited by 7
  5. 2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuningcited by 4
  6. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  7. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  8. 2026MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicinecited by 4
  9. 2025Enhancing Multi-modal Models with Heterogeneous MoE Adapters for Fine-tuningcited by 3
  10. 2025μ-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Expertscited by 3
  11. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  12. 2024Take Only What You Need: Rank Minimization as an Implicit Forgetting Regularizer in Continual Learningcited by 2
7 more
  1. 2025Compositional Subspace Representation Fine-tuning for Adaptive Large Language Modelscited by 2
  2. 2025Distillation-Supervised Convolutional Low-Rank Adaptation for Efficient Image Super-Resolutioncited by 2
  3. 2025Mix-of-Language-Experts Architecture for Multilingual Programmingcited by 2
  4. 2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1
  5. 2026LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modulescited by 1
  6. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  7. 2025GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Higher Layers Need More LoRA Experts
  2. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  3. 2026Parameter-Efficient Fine-Tuning with Learnable Rank
  4. 2025Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
  5. 2026MoE2-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
  6. 2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
  7. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  8. 2026ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition