Data to 7 Oct 2026

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

arXiv 2312.09979 ↗First posted 15 Dec 2023cs.CLcited by 82

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Supervised fine-tuning (SFT) is a crucial step for large language models (LLMs), enabling them to align with human instructions and enhance their capabilities in downstream tasks. Increasing instruction data substantially is a direct solution to align the model with a broader range of downstream tasks or notably improve its performance on a specific task. However, we find that large-scale increases in instruction data can damage the world knowledge previously stored in LLMs. To address this challenge, we propose LoRAMoE, a novelty framework that introduces several low-rank adapters (LoRA) and integrates them by using a router network, like a plugin version of Mixture of Experts (MoE). It freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks, to alleviate world knowledge-edge forgetting. Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks, while maintaining the world knowledge stored in the LLM.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 32 of its 54 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

32 of 54 references.

References · 32

2022Training language models to follow instructions with human feedbackcited by 15,712
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2015Character-level Convolutional Networks for Text Classificationcited by 2,293
2019Language Models as Knowledge Bases?cited by 1,922
2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308

20 more

2023LIMA: Less Is More for Alignmentcited by 1,080
2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
2021Scaling Vision with Sparse Mixture of Expertscited by 740
2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
2022A Review on Language Models as Knowledge Basescited by 156
2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
2019Answering Complex Open-domain Questions Through Iterative Query Generationcited by 60
2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
2019Episodic Memory Reader: Learning What to Remember for Question Answering from Streaming Datacited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 32; 2026: 23.

82 papers.

Sort:

By area:

2024A Survey on Mixture of Experts in Large Language Modelscited by 233
2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 118
2024Higher Layers Need More LoRA Expertscited by 101
2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 99
2024MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuningcited by 89
2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
2024A Survey on LoRA of Large Language Modelscited by 67
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48

38 more

2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 41
2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
2025Omni-Effects: Unified and Spatially-Controllable Visual Effects Generationcited by 27
2025OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuningcited by 20
2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-designcited by 18
2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
2024SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentationcited by 12
2025Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learningcited by 12
2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 12
2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 10
2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 10
2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
2024PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learningcited by 8
2025AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projectionscited by 8
2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
2024CITI: Enhancing Tool Utilizing Ability in Large Language Models without Sacrificing General Performancecited by 6
2024Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Mergingcited by 6
2025MokA: Multimodal Low-Rank Adaptation for MLLMscited by 6
2026Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptationcited by 6
2026MAU-GPT: Enhancing Multi-type Industrial Anomaly Understanding via Anomaly-aware and Generalist Experts Adaptationcited by 6
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
2024Selected Languages are All You Need for Cross-lingual Truthfulness Transfercited by 4
2024MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learningcited by 4
2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuningcited by 4
2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 4
2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
2026CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Mergingcited by 4
2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
2025FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Expertscited by 3
2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
2025Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAscited by 3

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way