Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix
Not on the map. Why?
Abstract (arXiv)
Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data. We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA (B=0): the gradient with respect to A is identically zero (partialL/partial A=0), so gradient masks select strictly from the B matrix whereas unrestricted random masks draw from both A and B. Such a comparison changes both the selection rule and the candidate parameter pool. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B. The effect is concentrated on the medical task, which contributes 80% of the aggregate; code and math move in the same direction, and legal shows no reliable difference. This advantage is consistent with a stability--plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting. On the two discrete-option tasks (Medical and Legal), the mean accuracy-BWT difference is not statistically distinguishable from zero.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 8 Oct 2026.
Builds on
29 of 34 references.
References · 29
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
- 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
- 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
- 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
17 more
- 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
- 2019Orthogonal Gradient Descent for Continual Learningcited by 369
- 2021Gradient Projection Memory for Continual Learningcited by 348
- 2024LoRA Learns Less and Forgets Lesscited by 328
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
- 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
- 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
- 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
- 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
- 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 17
- 2025Reasoning-Enhanced Domain-Adaptive Pretraining of Multimodal Large Language Models for Short Video Content Governancecited by 16
- 2025Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansioncited by 15
- 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 11
- 2024IPS: In-Prompt Process Supervision for Short Video Content Moderationcited by 7
- 2026The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVRcited by 3
- 2026Attribution-Guided Continual Learning for Large Language Modelscited by 1
- 2026When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuningcited by 1
Built on by
Not cited yet.