mlsanityData to 14 Oct 2026

Is Parameter Collision Hindering Continual Learning in LLMs?

arXiv 2410.10179 ↗First posted 14 Oct 2024cs.LG cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) often suffer from catastrophic forgetting when learning multiple tasks sequentially, making continual learning (CL) essential for their dynamic deployment. Existing state-of-the-art (SOTA) methods, such as O-LoRA, typically focus on constructing orthogonality tasks to decouple parameter interdependence from various domains.In this paper, we reveal that building non-collision parameters is a more critical factor in addressing CL challenges. Our theoretical and experimental analyses demonstrate that non-collision parameters can provide better task orthogonality, which is a sufficient but unnecessary condition. Furthermore, knowledge from multiple domains will be preserved in non-collision parameter subspaces, making it more difficult to forget previously seen data. Leveraging this insight, we propose Non-collision Low-Rank Adaptation (N-LoRA), a simple yet effective approach leveraging low collision rates to enhance CL in LLMs. Experimental results on multiple CL benchmarks indicate that N-LoRA achieves superior performance (+2.9), higher task orthogonality (*4.1 times), and lower parameter collision (*58.1 times) than SOTA methods.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

26 of 41 references.

References · 26

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2016Learning without Forgettingcited by 2,496
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  12. 2021Learning to Prompt for Continual Learningcited by 944
14 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  3. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  5. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  6. 2021Gradient Projection Memory for Continual Learningcited by 348
  7. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  8. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  9. 2019Episodic Memory in Lifelong Language Learningcited by 188
  10. 2024Continual Learning for Large Language Models: A Surveycited by 147
  11. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 104
  12. 2023Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detectioncited by 32
  13. 2020A Concise yet Effective model for Non-Aligned Incomplete Multi-view and Missing Multi-label Learningcited by 19
  14. 2024Sparse High Rank Adapterscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 8.

15 papers.

Sort:

By area:

  1. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  2. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  3. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
  4. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  5. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 11
  6. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  7. 2025Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Modelscited by 7
  8. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuningcited by 5
  9. 2026Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRAcited by 3
  10. 2026Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRAcited by 2
  11. 2026Continual Fine-Tuning of Large Language Models via Program Memorycited by 1
  12. 2026SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Modelscited by 1
3 more
  1. 2026Rethinking Transfer in Continual Learning: A Replay-Based Realisationcited by 1
  2. 2026Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learningcited by 1
  3. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 1