Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes
Not on the map. Why?
Abstract (arXiv)
Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9%+/-4.8% average forgetting, whereas standard LoRA (r=8, query/value modules) yields 0.6%+/-1.4% (paired t-test, p=0.002, Cohen's d_s=3.12). Task-level analyses confirm this reduction is not merely an aggregate effect. Secondary experiments on RoBERTa-base show the same pattern, and the strongest EWC baseline remains at 15.5%+/-1.4% forgetting. A six-task extension reveals that low average forgetting can hide strong task-level heterogeneity. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95%, with classifier-only and shallow-adapter baselines approaching LoRA. Companion task-similarity probes in GPT-2 and RoBERTa show the same directional story: frozen-backbone regimes preserve higher inter-task similarity than full fine-tuning, gradual unfreezing weakens stability, and full fine-tuning exhibits its clearest divergence at the final transformer layer. These results support a restrained mechanistic interpretation: LoRA helps largely because backbone freezing preserves a more stable shared feature scaffold. We position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 29 Mar 2026.
Builds on
19 of 22 references.
References · 19
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2016Learning without Forgettingcited by 2,496
- 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2016Progressive Neural Networkscited by 1,361
- 2021Learning to Prompt for Continual Learningcited by 944
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
- 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
7 more
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
- 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
- 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
- 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 19
- 2025LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learningcited by 18
- 2026Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptationcited by 4
Built on by
Not cited yet.