Data to 7 Oct 2026

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

arXiv 2402.12851 ↗First posted 20 Feb 2024cs.CLcited by 54

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Fine-tuning is often necessary to enhance the adaptability of Large Language Models (LLM) to downstream tasks. Nonetheless, the process of updating billions of parameters demands significant computational resources and training time, which poses a substantial obstacle to the widespread application of large-scale models in various scenarios. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prominent paradigm in recent research. However, current PEFT approaches that employ a limited set of global parameters (such as LoRA, which adds low-rank approximation matrices to all weights) face challenges in flexibly combining different computational modules in downstream tasks. In this work, we introduce a novel PEFT method: MoELoRA. We consider LoRA as Mixture of Experts (MoE), and to mitigate the random routing phenomenon observed in MoE, we propose the utilization of contrastive learning to encourage experts to learn distinct features. We conducted experiments on 11 tasks in math reasoning and common-sense reasoning benchmarks. With the same number of parameters, our approach outperforms LoRA significantly. In math reasoning, MoELoRA achieved an average performance that was 4.2% higher than LoRA, and demonstrated competitive performance compared to the 175B GPT-3.5 on several benchmarks.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 39 of its 50 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

39 of 50 references.

References · 39

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2020Scaling Laws for Neural Language Modelscited by 6,411
2022Large Language Models are Zero-Shot Reasonerscited by 4,323
2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769

27 more

2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
2021Knowledge Neurons in Pretrained Transformerscited by 590
2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
2016Solving General Arithmetic Word Problemscited by 353
2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
2021Hash Layers For Large Sparse Modelscited by 236
2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
2019On the Measure of Intelligencecited by 129
2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 27; 2026: 13.

54 papers.

Sort:

By area:

2024A Survey on Mixture of Experts in Large Language Modelscited by 233
2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
2024LoRA Soups: Merging LoRAs for Practical Skill Composition Taskscited by 49
2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
2025Advancing Expert Specialization for Better MoEcited by 37
2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
2026Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuningcited by 15
2025Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Expertscited by 13
2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
2025Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Losscited by 11
2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10

38 more

2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
2025Ensembles of Low-Rank Expert Adapterscited by 9
2024SEE-DPO: Self Entropy Enhanced Direct Preference Optimizationcited by 7
2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
2025Why Do More Experts Fail? A Theoretical Analysis of Model Mergingcited by 6
2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorationscited by 6
2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
2025Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognitioncited by 5
2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 5
2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 4
2025BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversitycited by 4
2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
2025HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRAcited by 4
2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAscited by 3
2025Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuningcited by 3
2024FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Modelscited by 2
2024VELoRA: A Low-Rank Adaptation Approach for Efficient RGB-Event based Recognitioncited by 2
2025CoRe^2: Collect, Reflect and Refine to Generate Better and Fastercited by 2
2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Modelscited by 2
2025Mixture of Routerscited by 2
2025Generative Representational Learning of Foundation Models for Recommendationcited by 2
2025Low-Rank Expert Merging for Multi-Source Domain Adaptation in Person Re-Identificationcited by 2
2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
2026TRACE: Capability-Targeted Agentic Trainingcited by 2
2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1
2026Sparse Spectral LoRA: Routed Experts for Medical VLMscited by 1
2026The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Modelscited by 1
2026Beyond GSD-as-Token: Continuous Scale Conditioning for Remote Sensing VLMscited by 1
2025Error-Aware Knowledge Distillation via Targeted Revision for Customer-Service Summarizationcited by 0
2026Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinationscited by 0
2026Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocationcited by 0
2026Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atomscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way