mlsanityData to 7 Oct 2026

Rehearsal-Free Modular and Compositional Continual Learning for Language Models

arXiv 2404.00790 ↗First posted 31 Mar 2024cs.LG cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Continual learning aims at incrementally acquiring new knowledge while not forgetting existing knowledge. To overcome catastrophic forgetting, methods are either rehearsal-based, i.e., store data examples from previous tasks for data replay, or isolate parameters dedicated to each task. However, rehearsal-based methods raise privacy and memory issues, and parameter-isolation continual learning does not consider interaction between tasks, thus hindering knowledge transfer. In this work, we propose MoCL, a rehearsal-free Modular and Compositional Continual Learning framework which continually adds new modules to language models and composes them with existing modules. Experiments on various benchmarks show that MoCL outperforms state of the art and effectively facilitates knowledge transfer.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

27 of 35 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2016Learning without Forgettingcited by 2,465
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2017Continual Learning with Deep Generative Replaycited by 1,152
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2018Experience Replay for Continual Learningcited by 1,074
15 more
  1. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  2. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  3. 2019Episodic Memory in Lifelong Language Learningcited by 188
  4. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  5. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  6. 2017HDLTex: Hierarchical Deep Learning for Text Classificationcited by 105
  7. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  8. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  9. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  10. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  11. 2024Continual Learning and Catastrophic Forgettingcited by 73
  12. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  13. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  14. 2023NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysiscited by 10
  15. 2023GradSim: Gradient-Based Language Grouping for Effective Multilingual Trainingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 7.

26 papers.

Sort:

By area:

  1. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  2. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 29
  3. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  4. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  5. 2024Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Modelscited by 11
  6. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
  7. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  8. 2024In-context Continual Learning Assisted by an External Continual Learnercited by 7
  9. 2025AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projectioncited by 7
  10. 2024Continual Learning Using Only Large Language Model Promptingcited by 5
  11. 2025Achieving Upper Bound Accuracy of Joint Training in Continual Learningcited by 4
  12. 2025Self-Evolving LLMs via Continual Instruction Tuningcited by 4
14 more
  1. 2024Learn it or Leave it: Module Composition and Pruning for Continual Learningcited by 3
  2. 2026ELLA: Efficient Lifelong Learning for Adapters in Large Language Modelscited by 3
  3. 2025HiCL: Hippocampal-Inspired Continual Learningcited by 2
  4. 2025Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettingscited by 2
  5. 2026Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuningcited by 2
  6. 2025Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregationcited by 1
  7. 2025Logits Replay + MoClip: Stabilized, Low-Cost Post-Training with Minimal Forgettingcited by 1
  8. 2025Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspacescited by 0
  9. 2025PPSEBM: An Energy-Based Model with Progressive Parameter Selection for Continual Learningcited by 0
  10. 2026Repurposing Adversarial Perturbations for Continual Learning: From Defense to Active Alignmentcited by 0
  11. 2026The Parts Are Greater Than the Sum: Automated Task Sequencing for Efficient Training of Multi-Policy LLMscited by 0
  12. 2026STAIL: Semantic Text-Anchored Incremental Learning for Medical Imaging via Large Language Modelscited by 0
  13. 2026Continual Learning in Transitioncited by 0
  14. 2026Popular Knowledge Propagates More Errors in LLM Knowledge Updatingcited by 0