Data to 7 Oct 2026

MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

arXiv 2402.11260 ↗First posted 17 Feb 2024cs.CL cs.AIcited by 63

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Adapting large language models (LLMs) to new domains/tasks and enabling them to be efficient lifelong learners is a pivotal challenge. In this paper, we propose MoRAL, i.e., Mixture-of-Experts augmented Low-Rank Adaptation for Lifelong Learning. MoRAL combines the multi-tasking abilities of MoE with the fine-tuning abilities of LoRA for effective life-long learning of LLMs. In contrast to the conventional approaches that use factual triplets as inputs MoRAL relies on simple question-answer pairs, which is a more practical and effective strategy for robust and efficient learning. Owing to new data settings, we introduce a new evaluation benchmark namely: Life Long Learning of LLM (5L-bench) encompassing a newly curated dataset of question-answer pairs, and a set of evaluation metrics for rigorous evaluation of MoRAL in open-book and closed-book settings. Experimental evaluation shows (i) LLMs learn fast in open-book settings with up to 30.15% improvement in "RA" for Phi-2-2.7B compared to closed-book (for models fine-tuned with MoRAL); (ii) MoRAL shows higher performance improvement for models with a greater number of parameters; (iii) MoRAL is robust to catastrophic forgetting offering better knowledge retention compared to baselines.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 60 of its 89 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

60 of 89 references.

Closest ancestors · 4

Same direction, newest first

Other references · 56

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2020Language Models are Few-Shot Learnerscited by 32,199
2022Training language models to follow instructions with human feedbackcited by 15,712
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
2016Overcoming catastrophic forgetting in neural networkscited by 4,718
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348

38 more

2022Mass-Editing Memory in a Transformercited by 976
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
2021Editing Factual Knowledge in Language Modelscited by 548
2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
2023Larger language models do in-context learning differentlycited by 386
2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
2023Studying Large Language Model Generalization with Influence Functionscited by 311
2023Effective Long-Context Scaling of Foundation Modelscited by 301
2021Time-Aware Language Models as Temporal Knowledge Basescited by 290
2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
2022A Review on Language Models as Knowledge Basescited by 156
2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
2018Memory Efficient Experience Replay for Streaming Learningcited by 128
2018Continuous Learning in Single-Incremental-Task Scenarioscited by 109
2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
2023MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRAcited by 80
2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
2018Memory-based Parameter Adaptationcited by 35
2023Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistancecited by 33
2023Time is Encoded in the Weights of Finetuned Language Modelscited by 31

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 21; 2026: 14.

63 papers.

Sort:

By area:

2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
2024A Survey on LoRA of Large Language Modelscited by 67
2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
2025Advancing Expert Specialization for Better MoEcited by 37
2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
2024A Survey of Small Language Modelscited by 31
2024Editable Concept Bottleneck Modelscited by 22

38 more

2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19
2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compressioncited by 18
2024LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-designcited by 18
2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Qualitycited by 18
2024Multilingual Large Language Models: A Systematic Surveycited by 18
2025LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routingcited by 16
2024A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoningcited by 15
2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 14
2025Understanding How Value Neurons Shape the Generation of Specified Values in LLMscited by 13
2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
2024Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMscited by 11
2024Leveraging Logical Rules in Knowledge Editing: A Cherry on the Topcited by 11
2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoningcited by 11
2024Understanding Reasoning in Chain-of-Thought from the Hopfieldian Viewcited by 10
2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
2024MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Languagecited by 9
2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
2024MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learningcited by 7
2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
2024CITI: Enhancing Tool Utilizing Ability in Large Language Models without Sacrificing General Performancecited by 6
2024Towards Multi-dimensional Explanation Alignment for Medical Classificationcited by 6
2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Expertscited by 5
2025LoRA-Gen: Specializing Large Language Model via Online LoRA Generationcited by 5
2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuningcited by 4
2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
2024Auto-selected Knowledge Adapters for Lifelong Person Re-identificationcited by 3
2026ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimationcited by 3
2024DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Modelscited by 2
2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuningcited by 2
2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Modelscited by 2
2026Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learningcited by 2
2026AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimizationcited by 2
2026Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routingcited by 2
2024AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approachcited by 1
2025Localized LoRA: A Structured Low-Rank Approximation for Efficient Fine-Tuningcited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way