mlsanityData to 7 Oct 2026

Orthogonal Subspace Learning for Language Model Continual Learning

arXiv 2310.14152 ↗First posted 22 Oct 2023cs.CL cs.LGcited by 296

Not on the map. Why?

Cited by 39 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

Key facts. Cited by 296 papers in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

33 of 44 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2016Learning without Forgettingcited by 2,465
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
21 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2018Experience Replay for Continual Learningcited by 1,074
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  6. 2021Learning to Prompt for Continual Learningcited by 938
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  9. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  10. 2021Gradient Projection Memory for Continual Learningcited by 338
  11. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  12. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  13. 2019Episodic Memory in Lifelong Language Learningcited by 188
  14. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 167
  15. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  16. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  17. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  18. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  19. 2023Investigating Forgetting in Pre-Trained Representations Through Continual Learningcited by 19
  20. 2022MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspectivecited by 17
  21. 2023Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Modelcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 57; 2025: 93; 2026: 144.

296 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  4. 2024Continual Learning for Large Language Models: A Surveycited by 144
  5. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  6. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  7. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  8. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  9. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  10. 2024On Large Language Model Continual Unlearningcited by 51
  11. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  12. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
38 more
  1. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  2. 2024SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Modelscited by 37
  3. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 35
  4. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  5. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  6. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 32
  7. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  8. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 29
  9. 2025HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Modelcited by 29
  10. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  11. 2024Rehearsal-Free Modular and Compositional Continual Learning for Language Modelscited by 26
  12. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 26
  13. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 23
  14. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  15. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  16. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  17. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  18. 2024More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMscited by 21
  19. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  20. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
  21. 2024PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learningcited by 19
  22. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  23. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  24. 2024Sparse High Rank Adapterscited by 17
  25. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  26. 2025Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuningcited by 16
  27. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  28. 2024FouRA: Fourier Low Rank Adaptationcited by 15
  29. 2024Is Parameter Collision Hindering Continual Learning in LLMs?cited by 15
  30. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  31. 2024Fast and Continual Knowledge Graph Embedding via Incremental LoRAcited by 14
  32. 2024Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalizationcited by 14
  33. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  34. 2025MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Mergingcited by 14
  35. 2025LLaVA-c: Continual Improved Visual Instruction Tuningcited by 14
  36. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  37. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  38. 2025Think Small, Act Big: Primitive Prompt Learning for Lifelong Robot Manipulationcited by 13

50 most cited of 296 All 296 (JSON)