JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
Not on the map. Why?
Abstract (arXiv)
Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 17 Apr 2026.
Builds on
35 of 48 references.
References · 35
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2023A Survey of Large Language Modelscited by 2,576
- 2016Learning without Forgettingcited by 2,496
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
23 more
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2016Progressive Neural Networkscited by 1,361
- 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
- 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
- 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2024Large Language Models: A Surveycited by 525
- 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
- 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
- 2019Orthogonal Gradient Descent for Continual Learningcited by 369
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
- 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
- 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
- 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
- 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
- 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
- 2019Episodic Memory in Lifelong Language Learningcited by 188
- 2024Continual Learning for Large Language Models: A Surveycited by 147
- 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
- 2024Unlocking Continual Learning Abilities in Language Modelscited by 23
- 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 6
- 2026ELLA: Efficient Lifelong Learning for Adapters in Large Language Modelscited by 3
Built on by
Not cited yet.