FGGM: Fisher-Guided Gradient Masking for Continual Learning
Not on the map. Why?
Abstract (arXiv)
Catastrophic forgetting impairs the continuous learning of large language models. We propose Fisher-Guided Gradient Masking (FGGM), a framework that mitigates this by strategically selecting parameters for updates using diagonal Fisher Information. FGGM dynamically generates binary masks with adaptive thresholds, preserving critical parameters to balance stability and plasticity without requiring historical data. Unlike magnitude-based methods such as MIGU, our approach offers a mathematically principled parameter importance estimation. On the TRACE benchmark, FGGM shows a 9.6% relative improvement in retaining general capabilities over supervised fine-tuning (SFT) and a 4.4% improvement over MIGU on TRACE tasks. Additional analysis on code generation tasks confirms FGGM's superior performance and reduced forgetting, establishing it as an effective solution.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 26 Jan 2026.
Builds on
23 of 28 references.
References · 23
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2016Learning without Forgettingcited by 2,465
- 2024Qwen2 Technical Reportcited by 2,076
- 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
- 2019Orthogonal Gradient Descent for Continual Learningcited by 361
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
11 more
- 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
- 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
- 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
- 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
- 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
- 2022Fine-tuned Language Models are Continual Learnerscited by 152
- 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
- 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
- 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
- 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
- 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
Built on by
Citing papers per year: 2026: 1.
1 paper.